2019-04-16 10:17:15 UTC - stefan: @stefan has joined the channel
----
2019-04-16 10:45:54 UTC - Samuel Sun: Hi all, for pulsar connector , we already
have a lot of implementation for different protocol, I wonder if we can have a
plugin to make source , pulsar topic/queue, sink together, works like a
pipeline thing, this plugin dose not focus the source or target protocol, but
can integrate kinds of source and targets implementation only by simple config.
----
2019-04-16 10:47:48 UTC - Ali Ahmed: @Samuel Sun are you about a e2e pipeline ?
----
2019-04-16 13:04:28 UTC - Samuel Sun: @Ali Ahmed not sure what you mean by e2e,
what I mean is we could this pipeline plugin config to compose a workflow of
data process, like
```
source(any protocol) + queue(non-persistent) + sink(any protocol), what this
connector focus is only source push + queue + sink pull without any specialized
protocol
```
----
2019-04-16 13:08:29 UTC - Samuel Sun: if it makes senses, I could try to make
one, it may focus on make a easier config to help engineer compose different
source/target without care about queue implement, just focus workflow of
business.
----
2019-04-16 13:10:17 UTC - Alexandre DUVAL: Hi, after cleaning my bookies using
metaformat and bookieformat (from our discussion with @David Kjerrumgaard), I
have the following issues:
```11:23:36.203 [bookkeeper-ml-workers-OrderedExecutor-3-0] ERROR
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[yo/functions/persistent/assignments] Failed to open ledger 1153: Error while
recovering ledger
11:23:36.203 [bookkeeper-ml-workers-OrderedExecutor-3-0] WARN
org.apache.pulsar.broker.service.BrokerService - Failed to create topic
<persistent://yo/functions/assignments>
org.apache.bookkeeper.mledger.ManagedLedgerException: Error while recovering
ledger
11:23:36.203 [bookkeeper-ml-workers-OrderedExecutor-3-0] ERROR
org.apache.pulsar.broker.service.ServerCnx - [/192.168.10.10:44740] Failed to
create topic <persistent://yo/functions/assignments>
java.util.concurrent.CompletionException:
org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException:
org.apache.bookkeeper.mledger.ManagedLedgerException: Error while recovering
ledger```
Or maybe it's due to errors on cluster metadata initialization
```13:06:12.488 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn -
EventThread shut down for session: 0x1014dd3a59a0007
Exception in thread "main"
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists for /namespace
at org.apache.zookeeper.KeeperException.create(KeeperException.java:122)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792)
at
org.apache.pulsar.PulsarClusterMetadataSetup.main(PulsarClusterMetadataSetup.java:178)```
I think it's the same problem
----
2019-04-16 13:11:35 UTC - Alexandre DUVAL: Something relative to ZK, and I'm
open to reset and clean my zk nodes
----
2019-04-16 13:39:34 UTC - Sijie Guo: +1 that’s a very useful feature. :+1:
----
2019-04-16 15:35:28 UTC - Devin G. Bost: Hi @Jerry Peng, yes, that's the fully
qualified name.
I was able to get functions to successfully create by providing the path to the
jar as the second parameter, like this. However, I'm getting some odd results
for sources and sinks.
----
2019-04-16 15:42:55 UTC - Sébastien de Melo: Hello guys, I have just pulled the
pulsar-all:2.3.1 Docker image and it seems that is has not been updated.
The script /pulsar/bin/apply-config-from-env.py does not contain the last
changes, whereas it does in pulsar:2.3.1.
----
2019-04-16 15:47:06 UTC - Sébastien de Melo: By the way, it would be nice to be
able to quickly see which version is running
----
2019-04-16 15:49:49 UTC - Matteo Merli: @Sébastien de Melo Let me check the
image
----
2019-04-16 15:50:06 UTC - Matteo Merli: The broker will report the version in
the logs when it starts
----
2019-04-16 15:50:28 UTC - Sébastien de Melo: OK good to know, thank you!
----
2019-04-16 15:58:49 UTC - Sébastien de Melo: I confirm, the brokers deployed
with pulsar-all image report "[main] INFO
org.apache.pulsar.broker.PulsarService - Starting Pulsar Broker service;
version: '2.3.0.0'" (and '2.3.1' with pulsar image)
----
2019-04-16 16:11:23 UTC - Devin G. Bost: For example, for a sink, I'm getting:
"org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException:
HTTP 500 Request failed."
----
2019-04-16 16:12:32 UTC - Devin G. Bost: For a different sink, I'm getting:
"org.apache.pulsar.client.admin.PulsarAdminException:
/tmp/pulsar-nar/functions6069595533781956123.tmp-unpacked/META-INF/MANIFEST.MF
(No such file or directory)"
I'm not sure where it's getting that directory from.
----
2019-04-16 16:15:02 UTC - Devin G. Bost: If I pass null as the second parameter
for sinks, I still get a 500 error for the first sink. For a different sink, I
get:
"Sink Package is not provided", which is more expected.
----
2019-04-16 16:31:06 UTC - Devin G. Bost: I'm wondering if it's possible for the
sink creation to fail if the configs are pointing to a host that doesn't exist.
----
2019-04-16 16:31:29 UTC - Manoj Manickath: @Manoj Manickath has joined the
channel
----
2019-04-16 16:35:17 UTC - Matteo Merli: Yes, I saw the problem. Rebuilding all
the images now
----
2019-04-16 16:35:21 UTC - Matteo Merli: Thanks for reporting this!
----
2019-04-16 17:56:36 UTC - Matteo Merli: @Sébastien de Melo Updated the images
+1 : Chris Bartholomew
----
2019-04-16 18:29:34 UTC - Jerry Peng: @Devin G. Bost I am confused at the code
you posted before. Are you trying to create or update a function or source?
----
2019-04-16 18:30:31 UTC - Jerry Peng: to create sources and sinks, the admin
API for those are different than the one for functions i.e
admin.source().create() and admin.since().create()
----
2019-04-16 18:33:51 UTC - Jerry Peng: the sink config you listed above seems
correct
----
2019-04-16 18:35:33 UTC - Jerry Peng: @Devin G. Bost what was different between
the sinks in which you got a 500 error back and “Sink Package is not provided”?
----
2019-04-16 19:30:15 UTC - Devin G. Bost: @Jerry Peng I apologize for the
confusion. I'm upserting functions, sinks, and sources.
For sources, I resolved one of the issues by setting the destinationTopicName
as the topicName in the SourceConfig constructor.
After that, I restarted my pulsar standalone docker container, and I stopped
seeing the 500 error.
Now, I'm seeing one issue that happens with only some of the sinks and sources,
and I'm seeing a different issue that happens with functions at a later point
in the bulk upsert operation.
----
2019-04-16 19:30:39 UTC - Devin G. Bost: The remaining issue with some of the
sinks and sources is that I'm getting an error like this:
`org.apache.pulsar.client.admin.PulsarAdminException:
/tmp/pulsar-nar/functions4700231300249613534.tmp-unpacked/META-INF/MANIFEST.MF
(No such file or directory)`
----
2019-04-16 19:31:26 UTC - Devin G. Bost: Here's more of the stack trace:
```
org.apache.pulsar.client.admin.PulsarAdminException:
/tmp/pulsar-nar/functions4700231300249613534.tmp-unpacked/META-INF/MANIFEST.MF
(No such file or directory)
at
org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:181)
at
org.apache.pulsar.client.admin.internal.SinkImpl.createSink(SinkImpl.java:125)
at com.overstock.deployment.manifest.Sink.upsert(Sink.java:58)
```
----
2019-04-16 19:31:50 UTC - Devin G. Bost: Here's another one:
```
org.apache.pulsar.client.admin.PulsarAdminException:
/tmp/pulsar-nar/functions1872994445727810475.tmp-unpacked/META-INF/MANIFEST.MF
(No such file or directory)
at
org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:181)
at
org.apache.pulsar.client.admin.internal.SourceImpl.createSource(SourceImpl.java:125)
at com.overstock.deployment.manifest.Source.upsert(Source.java:53)
```
----
2019-04-16 19:32:12 UTC - Devin G. Bost: I'm investigating what's different for
these cases because I don't get the error for all of the sinks and sources.
----
2019-04-16 19:33:57 UTC - Devin G. Bost: @Jerry Peng
It appears that the issue _only_ occurs with Kafka sinks and Kafka sources. It
also appears that I'm getting the issue on _every_ Kafka sink and _every_ Kafka
source.
----
2019-04-16 19:41:52 UTC - Jerry Peng: @Devin G. Bost how are you creating the
kafka sources/sinks. Are you submitting the NAR file or are you submitting as
a built-in connector?
----
2019-04-16 19:44:34 UTC - Jerry Peng: and which Kafka source/sink are you using?
----
2019-04-16 21:59:12 UTC - Jacob O'Farrell: @Sijie Guo If I don't really care
about the data in topics, would I essentially be able to just do a rolling
restart of the bookies with the new disk configurations?
----
2019-04-16 22:20:28 UTC - Anthony Corletti: @Anthony Corletti has joined the
channel
----
2019-04-16 23:27:01 UTC - Devin G. Bost: Here's a screenshot of how I'm calling
the Java Admin API to create/update one of the problematic Kafka sinks. You can
see the values being passed in the sinkConfig object.
----
2019-04-16 23:27:11 UTC - Devin G. Bost: @Jerry Peng Does that answer your
question?
----
2019-04-16 23:27:51 UTC - Devin G. Bost: Seems to be working now.
----
2019-04-16 23:28:09 UTC - Jerry Peng: @Devin G. Bost so you are submitting your
own implementation of the Kafka sink?
----
2019-04-16 23:29:12 UTC - Jerry Peng: I am asking because you are submitting
your own JAR
----
2019-04-16 23:29:48 UTC - Devin G. Bost: I'm probably setting the values of the
sinkConfig in a misleading way because we're supposed to be using the built-in
one. Should I try removing the Jar parameter and see if it works?
----
2019-04-16 23:38:26 UTC - Jerry Peng: You can either submit the connectors in
via built-in way if you have built-in connector configured on the broker/worker
or submit the connector NAR file. We do also support submitting uber JARs for
connectors as well
----
2019-04-16 23:38:36 UTC - Jerry Peng: but I would recommend built-in or NAR
----
2019-04-16 23:39:16 UTC - Jerry Peng: you can download all the NARs for
connectors here:
<http://pulsar.apache.org/en/download/>
----
2019-04-17 00:37:38 UTC - Devin G. Bost: Thanks for the info. I'll need to look
into this further.
----
2019-04-17 00:42:40 UTC - Devin G. Bost: If I want to use the built-in one,
what do I need to pass to the Java Admin API to use it? Is that something that
is set on the SourceConfig or SinkConfig object?
----
2019-04-17 01:42:13 UTC - Sijie Guo: I am not sure how k8 handles the new disk
configuration change. Does it delete the volume and create a new one? Or does
it just expand the size.
If it is the later, you are good.
If it is the former, there will be a problem - bookie has a strict cookie
validation mechanism to ensure data consistency. If a bookie’s disk is removed
without properly decommission, the bookie will fail the cookie validation.
Since the cookie stored locally is removed when the disk was removed, but the
cookie stored in zk is still there.
So a better approach will be creating a bookie job with a new disk
configuration with a new job name, and deleting the old bookie job.
----
2019-04-17 01:43:28 UTC - Jacob O'Farrell: Sorry - I'm not sure I understand
what you mean by "bookie job"
----
2019-04-17 01:46:43 UTC - Sijie Guo: Ah, I mean a new bookkeeper service with a
new service name, and a new bookkeeper StatefulSet with a new name. The idea is
to make the new bookie have a new hostname with a new disk configuration.
----
2019-04-17 01:48:31 UTC - Jacob O'Farrell: What/where would I need to update to
make everything else aware of the new bookies?
----
2019-04-17 01:55:09 UTC - Jacob O'Farrell: P.s. thank you for your help
----
2019-04-17 01:56:48 UTC - Sijie Guo: Just make sure you are using the same
zookeeper servers. That’s all the steps you need. The idea is to make sure the
bookies are started with new configuration and new bookie id (e.g new bookie
hostname in your k8).
----
2019-04-17 05:50:08 UTC - Jacob O'Farrell: Hey all, wondering if anyone has any
experience decommissioning bookies - I believe I've successfully begun the
process, however the count of ledgers that need to be re-replicated doesn't
appear to changing... Is this expected behaviour?
<https://gyazo.com/3874dd3dc51659f5cf757d166f028b5a>
----
2019-04-17 08:49:42 UTC - Marc Le Labourier: We are still seeing Starting
Pulsar Broker service; version: ‘2.3.0.0’ with apachepulsar/pulsar-all:2.3.1.
Might be that kubernetes did not pull the latest images …
----
2019-04-17 08:58:49 UTC - Marc Le Labourier: It seems to be good now. Thanks
@Matteo Merli, @Devin G. Bost.
----
