2019-05-13 09:16:33 UTC - Alexandre DUVAL: Hi, when you apply a schema to a
topic, then you add a function using this topic as input. The process function
signature looks like (<SchemaType> input, Context context)?
----
2019-05-13 09:25:35 UTC - Alexandre DUVAL: I see options ```
--custom-schema-inputs
The map of input topics to Schema class names (as a JSON string)
--custom-serde-inputs
The map of input topics to SerDe class names (as a JSON string)
``` on pulsar-admin functions create, but do you have a quick example of the
full usage?
----
2019-05-13 09:44:14 UTC - bhagesharora: Hello everyone, How
consumer.seek(msg_id) method is working in case of retention policy . If we set
the time limit 5m or size flag 5M ??
----
2019-05-13 12:00:32 UTC - Justin:
----
2019-05-13 12:06:41 UTC - Justin: This is my domain class:
public class PulsarMqData<T> implements Serializable {
private static final long serialVersionUID = -1;
private T data;
private int count = 0;
private long ct = 0L;
private long rt = 0L;
private long pct = 0L;
private long prt = 0L;
//getter setter
}
----
2019-05-13 12:35:40 UTC - Alexandre DUVAL: My topic schema is
`{"version":2,"type":"STRING","timestamp":0,"data":"","properties":{}}`, but I
got ```12:30:07.109 [pulsar-io-21-4] WARN
org.apache.pulsar.broker.service.ServerCnx - [/192.168.10.15:36928] Got
exception BufferUnderflowException : null
java.nio.BufferUnderflowException: null
at java.nio.Buffer.nextGetIndex(Buffer.java:506) ~[?:1.8.0_192]
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:412)
~[?:1.8.0_192]
at
org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.versionFromBytes(BookkeeperSchemaStorage.java:169)
~[org.apache.pulsar-pulsar-broker-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at
org.apache.pulsar.broker.service.schema.SchemaRegistryServiceImpl.versionFromBytes(SchemaRegistryServiceImpl.java:128)
~[org.apache.pulsar-pulsar-broker-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at
org.apache.pulsar.broker.service.ServerCnx.handleGetSchema(ServerCnx.java:1328)
~[org.apache.pulsar-pulsar-broker-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at
org.apache.pulsar.common.api.PulsarDecoder.channelRead(PulsarDecoder.java:299)
~[org.apache.pulsar-pulsar-common-2.4.0-SNAPSHOT.jar:2.4.0-SNAPSHOT]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:433)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:330)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
``` on brokers node when I start my functions. It's all about String. Consume
works well with random pulsar client.
----
2019-05-13 12:47:36 UTC - Alexandre DUVAL: It's not taking the highest version
by default?
----
2019-05-13 15:23:10 UTC - Devin G. Bost: Any updates on this? We're scheduled
to go to beta in two days, and we need 2.3.2 for our incremental update code.
----
2019-05-13 15:33:26 UTC - Nathan Linebarger: @Nathan Linebarger has joined the
channel
----
2019-05-13 16:58:06 UTC - Matteo Merli: @bhagesharora When trying to seek to a
message id that was already deleted, it will get you to the earliest available
message
----
2019-05-13 19:06:26 UTC - Thor Sigurjonsson: @Devin G. Bost Permissions on
/data/provisioning causing bookie to not come up?
----
2019-05-13 19:07:07 UTC - Devin G. Bost: Permissions are `rwxrwxrw-` with me as
the owner.
----
2019-05-13 19:10:58 UTC - Thor Sigurjonsson: does /data exist in the container?
----
2019-05-13 19:12:48 UTC - Thor Sigurjonsson: You doing this on macos?
----
2019-05-13 19:19:49 UTC - Devin G. Bost: Yes and yes.
----
2019-05-13 20:05:53 UTC - Devin G. Bost: I accidentally deleted my thread.
Today is just one of those days.
----
2019-05-13 20:09:47 UTC - Devin G. Bost: I was missing the `-v
$PWD/data:/pulsar/data` part of the docker command. That's what the problem was.
This works:
```
docker run -it \
-p 6650:6650 \
-p 8080:8080 \
-v $PWD/data:/pulsar/data \
-v /data/provisioning:/data/provisioning \
apachepulsar/pulsar-standalone \
bin/pulsar standalone
```
----
2019-05-14 00:10:04 UTC - Devin G. Bost: What does it mean when I get `Reason:
Function in trigger function is not ready` when I try to run `bin/pulsar-admin
functions trigger` on my Pulsar function?
----
2019-05-14 00:48:07 UTC - Devin G. Bost: I figured out that there was a problem
with how it was created.
----
2019-05-14 01:01:13 UTC - Devin G. Bost: I'm attempting to use a Pulsar
function for continuous deployment to construct other Pulsar functions via the
Java Admin API.
When I send it the config data for a Pulsar component that I want it to create
through the Java Admin API, I'm getting this error:
```
00:47:00.452 [pulsar-client-io-70-1] INFO
org.apache.pulsar.client.impl.ProducerImpl -
[<persistent://ops/deployment/incrementalUpdates>] [standalone-0-3] Closed
Producer
org.apache.pulsar.client.admin.PulsarAdminException: A MultiException has 2
exceptions. They are:
1. java.lang.NoSuchMethodError:
org.apache.pulsar.common.util.ObjectMapperFactory.create()Lorg/apache/pulsar/shade/com/fasterxml/jackson/databind/ObjectMapper;
2. java.lang.IllegalStateException: Unable to perform operation: create on
org.apache.pulsar.client.admin.internal.JacksonConfigurator
at
org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:187)
at
org.apache.pulsar.client.admin.internal.TenantsImpl.getTenants(TenantsImpl.java:50)
at
com.overstock.dataeng.pulsar.deployment.manifest.Tenant.existsInPulsar(Tenant.java:119)
. . .
```
----
2019-05-14 01:13:12 UTC - Devin G. Bost: I checked the versions of the
dependencies in my function, and they match Pulsar's dependency versions. (I'm
using Pulsar 2.3.1.)
----
2019-05-14 01:18:25 UTC - Devin G. Bost: It seems surprising that the
`getTenants` call would result in this exception.
----
2019-05-14 01:21:26 UTC - Jerry Peng: @Devin G. Bost are you calling getTenant
within your function?
----
2019-05-14 01:21:31 UTC - Devin G. Bost: Yes.
----
2019-05-14 01:21:54 UTC - Devin G. Bost: @Jerry Peng It's that line in the
stacktrace:
`com.overstock.dataeng.pulsar.deployment.manifest.Tenant.existsInPulsar(Tenant.java:119)`
----
2019-05-14 01:22:30 UTC - Jerry Peng: @Devin G. Bost you included
pulsar-client-admin in the pom as a dependency of the function?
----
2019-05-14 01:22:48 UTC - Jerry Peng: also are you building an uber jar for the
function?
----
2019-05-14 01:23:43 UTC - Jerry Peng: based on the stacktrace, a probable cause
is that the pular-client-admin JAR/Classes is not included in the function JAR
----
2019-05-14 01:24:09 UTC - Devin G. Bost: I'll check on that. Yes, it's a rather
large Jar.
----
2019-05-14 01:25:00 UTC - Jerry Peng: @Devin G. Bost you can inspect the
contents of your JAR and make sure class
“org/apache/pulsar/shade/com/fasterxml/jackson/databind/ObjectMapper” is in it.
----
2019-05-14 01:25:03 UTC - Devin G. Bost: This is in my POM:
```
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client-admin</artifactId>
<version>${pulsar.version}</version>
</dependency>
```
----
2019-05-14 01:25:10 UTC - Devin G. Bost: Sure thing.
----
2019-05-14 01:25:37 UTC - Jerry Peng: Thats good but make sure you are building
the function JAR as an uber JAR that includes this dependency
----
2019-05-14 01:26:13 UTC - Devin G. Bost: Like this?
```
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.1.1</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>assemble-all</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
```
----
2019-05-14 01:28:26 UTC - Devin G. Bost: Here's what I found in my Jar in the
directory org/apache/pulsar/shade/com/fasterxml/jackson/databind/
----
2019-05-14 01:29:12 UTC - Devin G. Bost: `ObjectMapper.class` is in there.
----
2019-05-14 01:34:29 UTC - Jerry Peng: @Devin G. Bost this is after you unzipped
the JAR?
----
2019-05-14 01:34:33 UTC - Devin G. Bost: Yes.
----
2019-05-14 01:36:05 UTC - Jerry Peng: Actually the exception indicates that
```org.apache.pulsar.common.util.ObjectMapperFactory``` class is not found
----
2019-05-14 01:36:15 UTC - Jerry Peng: please check for that class
----
2019-05-14 01:36:38 UTC - Devin G. Bost: Oh, good catch. I wasn't sure how to
interpret the exception message.
----
2019-05-14 01:36:42 UTC - Devin G. Bost: I'll check.
----
2019-05-14 01:37:56 UTC - Devin G. Bost: @Jerry Peng Sure enough,
`ObjectMapperFactory.class` is in `/org/apache/pulsar/common/util/` in the
unpacked jar.
----
2019-05-14 01:41:04 UTC - Devin G. Bost: I just re-built the jar and
double-checked, and both of those classes are present.
----
2019-05-14 01:46:52 UTC - Jerry Peng: I think I know the issue. There is
probably a class/shading conflict with java-instance.jar. ObjectMapperFactory
is included with java-instance.jar but its looking for
org/apache/pulsar/functions/runtime/shaded/com/fasterxml/jackson/databind/ObjectMapper.
While pulsar-client-admin is looking for another version. I will have to
shaded the pulsar-common dependency in java-instance.jar. Users don’t usually
use pulsar-admin API inside functions. We can still support it
----
2019-05-14 01:47:28 UTC - Jerry Peng: @Devin G. Bost I will make a PR to fix
the shading to make this work
----
2019-05-14 01:48:09 UTC - Devin G. Bost: @Jerry Peng Thank you very much for
supporting this use case! This is a key part of our CI/CD pipeline architecture
and plans.
----
2019-05-14 01:48:44 UTC - Jerry Peng: @Devin G. Bost np shouldn’t be a
difficult fix. I will have a PR up shortly
----
2019-05-14 01:50:36 UTC - Devin G. Bost: Thanks @Jerry Peng! We're actually
scheduled to go to beta on Wednesday, so we will eagerly look forward to the
fix... :sweat_smile:
----
2019-05-14 01:53:24 UTC - Jerry Peng: @Devin G. Bost as an alternative to
consider instead of using the pulsar-admin API you can use a HTTP client and
call out the the Pulsar’s REST interface directly which is what the
pulsar-admin API is doing underneath anyways.
----
2019-05-14 01:53:45 UTC - Jerry Peng: Obviously that would be more burdensome
and you will have to for the http requests
----
2019-05-14 01:54:34 UTC - Devin G. Bost: Thanks for the suggestion. However, at
this point, that would be a bit of a re-write, so I think we will need to
explore another workaround until you guys are able to ship the fix.
----
2019-05-14 02:04:19 UTC - Devin G. Bost: @Jerry Peng Do you think it would work
if we created a consumer with the Pulsar client library?
----
2019-05-14 02:11:49 UTC - Matteo Merli: Yes, if you just want to ensure a
subscription is there, you can just subscribe and immediately close the
consumer.
----
2019-05-14 02:12:32 UTC - Devin G. Bost: Sounds good. Thanks for the guidance.
----
2019-05-14 04:38:39 UTC - Sanjeev Kulkarni: @Devin G. Bost I would highly
recommend against using pulsar admin library inside your function. It’s not
really meant to be a general purpose library
----
2019-05-14 04:39:04 UTC - Sanjeev Kulkarni: The idea of using a http interface
or some other simpler rest interface is what I would recommend as well
----
2019-05-14 07:20:40 UTC - Jerry Peng: @Devin G. Bost ya I would also recommend
that as well
----
2019-05-14 08:47:59 UTC - bhagesharora: Ok.. Thanks for your response, Can you
please explain where we need to configure or mention time limit or size limit.
even I tried out with pulsar-admin command and changing in broker.conf file. so
After the publishing message in consumer acknowledge message is not coming but
when I am using reader interface(rewind process) message is still coming.
----
