2019-03-14 09:12:23 UTC - bhagesharora: Hello Everyone
I have seen one of example in pulsar SQL, I have doubt in same example
I am following URl for the reference
<https://pulsar.apache.org/docs/en/sql-getting-started/>
here we have AvroSchema see below
public static void main(String[] args) throws Exception {
PulsarClient pulsarClient =
PulsarClient.builder().serviceUrl("<pulsar://localhost:6650>").build();
Producer<Foo> producer =
pulsarClient.newProducer(AvroSchema.of(Foo.class)).topic("test_topic").create();
I didn't find any java class or library/jar with the name of AvroSchema
so here What is the meaning of AvroSchema, It's a predefine Java class or we
need to create a AvroSchema class ??
----
2019-03-14 09:19:54 UTC - pingle wang: @bhagesharora fix Schema.AVRO(Foo.class)
o try
----
2019-03-14 09:29:25 UTC - bhagesharora: @pingle wang Producer<Foo>
producer =
pulsarClient.newProducer(Schema.AVRO(Foo.class)).topic("test_topic").create();
Now error is showing "The method AVRO(Class<Test.Foo>) is undefined for
the type Schema"
----
2019-03-14 09:31:27 UTC - Sijie Guo: what version of pulsar are you using?
----
2019-03-14 09:32:48 UTC - bhagesharora: 2.0.1
----
2019-03-14 09:33:15 UTC - bhagesharora: 2.0.1-incubating
----
2019-03-14 09:34:35 UTC - Sijie Guo: can you use 2.3.0?
+1 : bhagesharora
----
2019-03-14 09:35:56 UTC - bhagesharora: ok, I'll try with 2.3.0 but AvroSchema
is a predefined class ??
----
2019-03-14 09:36:31 UTC - Sijie Guo: AvroSchema is a class for people to use
Avro schema
----
2019-03-14 09:37:21 UTC - Ganga Lakshmanasamy: yes @Sanjeev Kulkarni both are
in same docker only.
----
2019-03-14 09:37:38 UTC - bhagesharora: so manually we need to create ??
----
2019-03-14 09:40:14 UTC - Sijie Guo: create ??
----
2019-03-14 09:43:55 UTC - bhagesharora: so manually we need to create/build a
AvroSchema class ??
----
2019-03-14 09:46:05 UTC - Sijie Guo: Schema.AVRO(Foo.class) will parse the
Foo.class and generate a schema for Foo.
----
2019-03-14 09:46:14 UTC - Sijie Guo: I am not sure if this answers your question
----
2019-03-14 09:51:32 UTC - bhagesharora: so it will work on 2.3.0 version ??
because in 2.0.1-incubation, when I am using above line which you have
mentioned, giving error.
----
2019-03-14 09:51:35 UTC - bhagesharora: Producer<Foo> producer =
pulsarClient.newProducer(Schema.AVRO(Foo.class)).topic("test_topic").create();
Now error is showing "The method AVRO(Class<Test.Foo>) is undefined for
the type Schema"
----
2019-03-14 09:56:52 UTC - Sijie Guo: are you using 2.3.0?
----
2019-03-14 09:57:46 UTC - Sijie Guo:
<http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/Schema.html#AVRO-java.lang.Class->
+1 : bhagesharora
----
2019-03-14 10:18:24 UTC - pingle wang: sorry, I make a mistake.
2.0.1-incubating version not has this method.
+1 : bhagesharora
----
2019-03-14 11:12:03 UTC - Alexandre DUVAL: @Sijie Guo
----
2019-03-14 11:18:16 UTC - Alexandre DUVAL: Am I right or I didn't understand
something :stuck_out_tongue:.
----
2019-03-14 11:19:21 UTC - Sijie Guo: I see. so I think most of the http client
strip out authentication header when following redirects for security concerns.
this is kind of a *bug*.
what we can do is for pulsar (admin) client to handle this direct and post the
request with authentication header to the right location when redirect happens.
but it doesn’t address the issues if you curl directly. we might have to
document this behavior
----
2019-03-14 11:19:29 UTC - Sijie Guo: does this make sense?
----
2019-03-14 11:20:58 UTC - Alexandre DUVAL: It makes sense, I was using
pulsar-admin yes.
----
2019-03-14 11:21:47 UTC - Alexandre DUVAL: Do you think we can have a
quick-fix? Or should I write a PR for it? :stuck_out_tongue:
----
2019-03-14 11:22:58 UTC - Sijie Guo: give me a few more mins
:slightly_smiling_face:
----
2019-03-14 11:23:25 UTC - Alexandre DUVAL: :heart:.
----
2019-03-14 11:23:28 UTC - Alexandre DUVAL: :stuck_out_tongue:
----
2019-03-14 11:39:46 UTC - Sijie Guo: diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/HttpClient.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/HttpClient.java
index 43e5bd7c4..11f49cba6 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/HttpClient.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/HttpClient.java
@@ -50,6 +50,9 @@ import org.asynchttpclient.ListenableFuture;
import org.asynchttpclient.Request;
import org.asynchttpclient.Response;
import org.asynchttpclient.channel.DefaultKeepAliveStrategy;
+import org.asynchttpclient.filter.FilterContext;
+import org.asynchttpclient.filter.FilterException;
+import org.asynchttpclient.filter.RequestFilter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -88,6 +91,22 @@ public class HttpClient implements Closeable {
return (response.status().code() / 100 != 5) &&
super.keepAlive(ahcRequest, request, response);
}
});
+ confBuilder.addRequestFilter(new RequestFilter() {
+ @Override
+ public <T> FilterContext<T>
filter(FilterContext<T> ctx) throws FilterException {
+ AuthenticationDataProvider authData = null;
+ try {
+ authData = authentication.getAuthData();
+ } catch (PulsarClientException e) {
+ throw new FilterException(“Failed to get authentication
data”, e);
+ }
+ for (Map.Entry<String, String> header :
authData.getHttpHeaders()) {
+ ctx.getRequest().getHeaders()
+ .set(header.getKey(), header.getValue());
+ }
+ return ctx;
+ }
+ });
if
(“https”.equals(serviceNameResolver.getServiceUri().getServiceName())) {
try {
@@ -110,7 +129,9 @@ public class HttpClient implements Closeable {
}
confBuilder.setEventLoopGroup(eventLoopGroup);
AsyncHttpClientConfig config = confBuilder.build();
- httpClient = new DefaultAsyncHttpClient(config);
+ httpClient = new DefaultAsyncHttpClient(
+ config
+ );
log.debug(“Using HTTP url: {}“, serviceUrl);
}
----
2019-03-14 11:39:59 UTC - Sijie Guo: I think something like this might be
working
----
2019-03-14 11:40:05 UTC - Sijie Guo: although I haven’t tried that
----
2019-03-14 11:40:13 UTC - Sijie Guo: :laughing:
----
2019-03-14 11:41:02 UTC - Sijie Guo: basically the idea is using a request
filter in the http client, we attached the authentication data when the filter
is executed.
----
2019-03-14 11:41:43 UTC - Alexandre DUVAL: Ok, I'll try.
----
2019-03-14 12:16:10 UTC - Alexandre DUVAL: @Sijie Guo hey, can you attach a
file of the diff, seems slack corrupts the text you provided :stuck_out_tongue:.
----
2019-03-14 12:19:03 UTC - Sijie Guo:
----
2019-03-14 12:20:23 UTC - Alexandre DUVAL: thx
----
2019-03-14 12:23:23 UTC - Alexandre DUVAL: building.. :slightly_smiling_face:
----
2019-03-14 12:24:04 UTC - Sijie Guo: lol I haven’t tested it
:slightly_smiling_face:
----
2019-03-14 12:24:07 UTC - Sijie Guo: good luck
----
2019-03-14 12:25:40 UTC - Alexandre DUVAL: I know, I will for us
:stuck_out_tongue:.
----
2019-03-14 13:55:39 UTC - Alexandre DUVAL: I did ` mvn install -DskipTests` but
here is more to do, to update the bin files? no?
----
2019-03-14 13:56:10 UTC - Alexandre DUVAL: I can see that they are not updated,
thanks do last modified file date.
----
2019-03-14 13:56:42 UTC - Alexandre DUVAL: @Sijie Guo
----
2019-03-14 14:05:49 UTC - Sébastien de Melo: Hi guys,
After some time (20 hours) our 2 brokers start crashing in a loop because of
the following error (related to the function worker):
----
2019-03-14 14:10:55 UTC - bhagesharora: Hello everyone,
I am trying to implement pulsar_sql
Using the below code I am able to create a producer
But what is the next step, I mean in presto what schema table/data will create?
where we can fire the query/In which object we can fire the query ?
public class Test {
public static class Foo {
private int field1 = 1;
private String field2;
private long field3;
}
public static void main(String[] args) throws Exception {
PulsarClient pulsarClient =
PulsarClient.builder().serviceUrl("<pulsar://localhost:6650>").build();
Producer<Foo> producer =
pulsarClient.newProducer(AvroSchema.of(Foo.class)).topic("test_topic").create();
for (int i = 0; i < 1000; i++) {
Foo foo = new Foo();
foo.setField1(i);
foo.setField2("foo" + i);
foo.setField3(System.currentTimeMillis());
producer.newMessage().value(foo).send();
}
producer.close();
pulsarClient.close();
}
}
----
2019-03-14 15:14:11 UTC - JAYARAM NAGARAJAN: @Karthikeyan Palanivelu @Sijie Guo
Thanks we were able to get past this step.. however got one another issue
while trying to produce, @Karthikeyan Palanivelu sending you a mail on this..
----
2019-03-14 15:22:59 UTC - Sijie Guo: if you run `mbn install -DskipTests`, you
can use `bin/pulsar-admin` to invoke the admin commands.
----
2019-03-14 15:30:53 UTC - Alexandre DUVAL: ok, so it doesn't work with the
patch too. I'll check why.
----
2019-03-14 15:31:03 UTC - Alexandre DUVAL: and if the error have changed
----
2019-03-14 15:47:06 UTC - Sijie Guo: I would suggest you checking out the
pulsar sql documentation - <http://pulsar.apache.org/docs/en/sql-overview/>
----
2019-03-14 15:47:21 UTC - Sijie Guo: getting started is a good section to start
with
----
2019-03-14 15:49:09 UTC - Sijie Guo: I don’t think that’s the right error. the
errors here more look like a side effect of two brokers crashed. I would
suggest looking into the logs and find the logs when the broker firs time
crashed.
----
2019-03-14 16:42:53 UTC - Alexandre DUVAL: It doesn't work, the header is not
added.
----
2019-03-14 16:43:46 UTC - Alexandre DUVAL: Maybe @Matteo Merli can have a look?
:stuck_out_tongue:
----
2019-03-14 16:44:02 UTC - Sébastien de Melo: Thanks for your answer, I have
already restarted the brokers but I will try to find the first error next time
----
2019-03-14 17:21:30 UTC - Sanjeev Kulkarni: did you login into the container
and see if the broker was running? Any logs
----
2019-03-14 21:22:13 UTC - Matteo Merli: Catching up now. So is this showing the
proxy is not forwarding the `Authorization` header from client?
heavy_check_mark : Alexandre DUVAL
----
2019-03-14 21:22:51 UTC - Matteo Merli: I’ll try to reproduce in an integration
test and look for a fix
----
2019-03-14 21:23:49 UTC - Matteo Merli: Yes, running with `-b 0` should disable
batching
----
2019-03-14 21:24:00 UTC - Matteo Merli: Are you still seeing the discrepancy ?
----
2019-03-14 21:48:07 UTC - Ravi Mahendrakar: @Ravi Mahendrakar has joined the
channel
----
2019-03-14 22:06:24 UTC - Alexandre DUVAL: Many thanks.
----
2019-03-15 00:37:03 UTC - JZT-静宝: @JZT-静宝 has joined the channel
----
2019-03-15 04:08:15 UTC - jia zhai: sorry for the delay. how about this one?
----
2019-03-15 04:23:44 UTC - bossbaby: When bookkeeper has enabled kerberos
authentication, how can the broker connect to the bookkeeper?
----
2019-03-15 05:35:03 UTC - jia zhai: @bossbaby broker need these setting, in
broker.conf:
```
# Authentication plugin to use when connecting to bookies
bookkeeperClientAuthenticationPlugin=
# BookKeeper auth plugin implementatation specifics parameters name and values
bookkeeperClientAuthenticationParametersName=
bookkeeperClientAuthenticationParameters=
```
----
2019-03-15 06:42:41 UTC - bossbaby: @jia zhai i did it but broker can't connect
bookie
this's my log:
<https://gist.github.com/tuan6956/8105ad88e04a3e51bb9384d5db1ea905>
----
2019-03-15 07:01:44 UTC - jia zhai: It contains errors like this:
```
11:38:37.942 [pulsar-ordered-OrderedExecutor-5-0-SendThread(localhost:2181)]
WARN org.apache.zookeeper.ClientCnxn - SASL configuration failed:
javax.security.auth.login.LoginException: No JAAS configuration section named
'Client' was found in specified JAAS configuration file:
'/Users/pro/Desktop/Apache_Pulsar/pulsar_authencation/apache-pulsar-2.3.0/bookie_jaas_client.conf'.
Will continue connection to Zookeeper server without SASL authentication, if
Zookeeper server allows it.
```
----
2019-03-15 07:02:54 UTC - jia zhai: ```
ERROR org.apache.bookkeeper.sasl.SASLClientAuthProvider - Error on SASL client
javax.security.sasl.SaslException: SASL/JAAS error
```
----
2019-03-15 07:04:27 UTC - jia zhai: you may need to figure out how to config it.
----
2019-03-15 07:20:31 UTC - bossbaby: So need to set up both zookeeper and broker?
----
2019-03-15 07:54:55 UTC - jia zhai: Seems the zookeeper error is a WARN, it
should be OK
----
2019-03-15 07:55:46 UTC - jia zhai: The main issue is that BookKeeper server
and BookKeeper client(Broker) is not set success.
----
2019-03-15 08:14:31 UTC - Kev Jackson: quick question: having checked the code,
it seems like service-discovery is currently hardcoded to use Zookeeper (in
that the this dependency is created in the start() method). We're heavy users
of HashiCorp Consul and it would make sense for us to use that for service
discovery in pulsar - checking the issues there is an open feature:
<https://github.com/apache/pulsar/issues/572> - which mentions BookKeeper as
depending on Zookeeper
----
2019-03-15 08:18:02 UTC - Kev Jackson: I was thinking of investigating using
consul for pulsar, but if there is a co-dependency that bookkeeper also
requires zookeeper then the scale of the change is larger than I expected
----
2019-03-15 08:19:01 UTC - Ali Ahmed: bk requires zk as well it is not simple to
move components to consul
----
2019-03-15 08:19:16 UTC - Maarten Tielemans: Both require Zookeeper. However,
both also share a good chunk of the development team to answer questions
----
2019-03-15 08:21:04 UTC - Ali Ahmed: people have tried projects like
<https://github.com/glerchundi/parkeeper>
----
2019-03-15 08:21:34 UTC - Ali Ahmed: to proxy requests but I don’t think any of
them have been tested in large production environments.
----
2019-03-15 08:30:46 UTC - Kev Jackson: ^ that's an interesting project, but
don't think it solves the issue I'd like to solve. I will take a dig into the
bookkeeper code to see how tightly coupled the projects are
----
2019-03-15 09:01:18 UTC - Sijie Guo: @Kev Jackson :
- bk has a much clear interfaces for pluggable metadata store. by default it is
zookeeper. there is an etcd support already. adding consul should be
straightforward
- pulsar requires some refactors to clean up the interfaces. so that we can
plugin the metadata part using etcd or consul.
I do receive same requests on using consul as the metadata store. this is
something we can prioritize on. the core pulsar/bookkeeper team can work on the
metadata abstraction, if any one is familiar with consul can help contribute a
consul plugin. that would be great.
----
