Kafka connector for Opentext Documentum xCP 2.2
My employer has Kafka for communicating some systems and I need to integrate my system too. I'm new to this subject, so I'll cut to the chase: Are there any Kafka connectors to use in Opentext Documentum xCP 2.2? Kind regards, Elber Esta mensagem e seus anexos são destinados exclusivamente ao(s) seu(s) destinatário(s) e, podem conter informações confidenciais e/ou legalmente privilegiadas. A divulgação, distribuição, reprodução ou qualquer forma de uso não autorizado de tais informações são proibidas e podem ser ilegais, sujeitando-se o responsável às penalidades cabíveis. O remetente utiliza o correio eletrônico no exercício do seu trabalho ou em razão dele, eximindo as empresas integrantes do Grupo de qualquer responsabilidade por utilização indevida. Caso não seja o destinatário desta mensagem, solicitamos a gentileza de notificar o remetente e eliminá-la imediatamente.
Embedded Kafka connector in data ingestion service
Hey there, I'm wondering if anyone has the need to use an embedded Kafka connector module. The goal we want to achieve is to avoid letting customers maintain a separate component when they stream data from their Kafka cluster to our service, so that they just need to provide the cluster connection information and we could internally manage all of the hard logic on cloud. It would be of our best interest if we don't need to have a standalone connector service either. I know this is not a usual solution, so just want to know if anyone has had to solve a similar problem, and what's your take on the trade-off, thanks!
Kafka connector for google BigQuery
Hi,
To stream data from kafka topic to Google BigQuery, i see some references
to
wepay-kafka-connect-bigquery-2.1.0
However, this sounds like being depreciated.
Currently we can stream directory through Spark Structured Streaming with
writeStream. \
outputMode('append'). \
option("truncate", "false"). \
foreachBatch(SendToBigQuery). \
This will use Spark-BigQuery API which isd pretty efficient .
I was looking at the Kafka connector for BigQuery and it appears that some
documents may not be the latest. For example
wepay-kafka-connect-bigquery-2.1.0
For example in the past Aerospike (NoSql database vendor) provided
connector for us and we just used the command
$KAFKA_HOME/bin/connect-standalone.sh \
/opt/aerospike-kafka-connect-sink/etc/connect-standalone.properties \
/opt/aerospike-kafka-connect-sink/etc/aerospike-sink.properties
The templates for those two property files were provided (plus jars) and we
updated them accordingly to get the stream going directly from Kafka to the
Aerospike table.
Can anyone point out to the latest version of kafka connector to Bigquery
documents?
Many thanks,
Mich
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
Re: [External] kafka connector distributed mode balance stratety
When clients connect the Kafka broker assigns each client a subset of the topic
partitions. When a client becomes unresponsive then the broker assigns those
partitions to another client.
The state of your connector task isn't going to be transferred but another
connector task will eventually get message the other task (the one that fails)
doesn't acknowledge..
-Dave
On 5/13/20, 10:42 PM, "wangl...@geekplus.com.cn"
wrote:
I want to know how kafka connector under distributed mode balance its task？
For example I have two connector instance： 192.168.10.2:8083,
192.168.10.3:8083
If one killed, the task can be transfered to another automatically without
any data loss?
When i use restful API curl "192.168.10.x:8083/.", the two instance
api is completely equivalent? Or i need I a cname for this two instances?
Thanks,
Lei
wangl...@geekplus.com.cn
This e-mail and any files transmitted with it are confidential, may contain
sensitive information, and are intended solely for the use of the individual or
entity to whom they are addressed. If you have received this e-mail in error,
please notify the sender by reply e-mail immediately and destroy all copies of
the e-mail and any attachments.
kafka connector distributed mode balance stratety
I want to know how kafka connector under distributed mode balance its task？ For example I have two connector instance： 192.168.10.2:8083, 192.168.10.3:8083 If one killed, the task can be transfered to another automatically without any data loss? When i use restful API curl "192.168.10.x:8083/.", the two instance api is completely equivalent? Or i need I a cname for this two instances? Thanks, Lei wangl...@geekplus.com.cn
Dose kafka connector rest aoi support multiple thread access?
Dear teams,
We use Kafka-1.1.0 connector to load data,And start a connector using
rest api by application.
Before we start a connector,we'll check it was not existed and then
create it. This was encapsulated in a Quartz job.And each connector
had a job.
We use spring resttemplate as below:
ResponseEntity response =
restTemplate.getForEntity(requestUrl, String.class);
But when the jobs running at the same time, it often throw two kinds
exception: "404 Not Found " and " 409 Conflict ". "404 Not Found " is a
normal exception,it meams the connector not existed.
I could not handle the " 409 Conflict " exception,it was happed when the
connector service is running,so I could not judge it was a connector
server exception or it was just the connector reject the request.
" 409 Conflict " also returned when I use curl to get connector list:
curl -X GET http://172.17.5.203:8083/connectors
{"error_code":409,"message":"Cannot complete request momentarily due to
stale configuration (typically caused by a concurrent config change)"}
Dear teams,how could I solve the problem?I'm not sure whether it was
caused by the connector rest api not support multiple thread access.
Thanks.
Re: Problem with Kafka Connector
Thanks robin Usually how many connectors can load on one worker to have independent kafka connect cluster Thanks Lakshman Sent from my iPhone > On 6 Dec 2018, at 9:49 PM, Robin Moffatt wrote: > > If the properties are not available per-connector, then you will have to > set them on the worker and have independent Kafka Connect clusters > delineated by connector requirements. So long as you configure the ports > not to clash, there's no reason these can't exist on the same host. > > > -- > > Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff > > >> On Wed, 5 Dec 2018 at 10:19, Федор Чернилин wrote: >> >> Hello! I have question. We have cluster with several connect workers. And >> we have many different connectors. We need to set for each connector its >> own settings, max.in.flight.requests.per.connection , partitioner.class, >> acks. But I have difficulties. How can I do that? Thanks
Re: Problem with Kafka Connector
Hello We have eight countries, each country have three connectors .total 24 connectors but we have one worker cluster , every time kafka connect worker need to restart In this case how we can manage , pls suggest Regards Lakshman > On 6 Dec 2018, at 9:49 PM, Robin Moffatt wrote: > > If the properties are not available per-connector, then you will have to > set them on the worker and have independent Kafka Connect clusters > delineated by connector requirements. So long as you configure the ports > not to clash, there's no reason these can't exist on the same host. > > > -- > > Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff > > >> On Wed, 5 Dec 2018 at 10:19, Федор Чернилин wrote: >> >> Hello! I have question. We have cluster with several connect workers. And >> we have many different connectors. We need to set for each connector its >> own settings, max.in.flight.requests.per.connection , partitioner.class, >> acks. But I have difficulties. How can I do that? Thanks
Re: Problem with Kafka Connector
If the properties are not available per-connector, then you will have to set them on the worker and have independent Kafka Connect clusters delineated by connector requirements. So long as you configure the ports not to clash, there's no reason these can't exist on the same host. -- Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff On Wed, 5 Dec 2018 at 10:19, Федор Чернилин wrote: > Hello! I have question. We have cluster with several connect workers. And > we have many different connectors. We need to set for each connector its > own settings, max.in.flight.requests.per.connection , partitioner.class, > acks. But I have difficulties. How can I do that? Thanks
Problem with Kafka Connector
Hello! I have question. We have cluster with several connect workers. And we have many different connectors. We need to set for each connector its own settings, max.in.flight.requests.per.connection , partitioner.class, acks. But I have difficulties. How can I do that? Thanks
Are there any connector configuration properties to limit the rate of messages that are sent from a Kafka cluster to a Kafka connector?
Hi, Is there a way to configure a Kafka connector (or a Kafka Connect server/cluster) to: 1. Receive a maximum of 1MB of data per second from a source Kafka cluster, or 2. Receive a maximum of 1000 records per second from a source Kafka cluster, or 3. Receive a maximum of 1MB of data and 1000 records per second from a source Kafka cluster Thanks, Behrang -- Best regards, Behrang Saeedzadeh
Re: Kafka connector throughput reduction upon avro schema change
What is your setting for schema.compatibility? I suspect the issue is probably that it is defaulting to NONE which would cause the connector to roll a new file when the schema changes (which will be frequent with data that is interleaved with different schemas). If you set it to BACKWARDS then the records would be properly projected and not require rolling files. Of course this assumes you are ok with records being projected to the latest schema. -Ewen On Thu, Jul 6, 2017 at 10:04 AM, Dave Hamilton wrote: > Bumping this. Has anyone here observed this in their Kafka connect > deployments? > > Thanks, > Dave > > > On 5/26/17, 1:44 PM, "Dave Hamilton" wrote: > > We are currently using the Kafka S3 connector to ship Avro data to S3. > We made a change to one of our Avro schemas and have noticed consumer > throughput on the Kafka connector drop considerably. I am wondering if > there is anything we can do to avoid such issues when we update schemas in > the future? > > This is what I believe is happening: > > > · The avro producer application is running on 12 instances. > They are restarted in a rolling fashion, switching from producing schema > version 1 before the restart to schema version 2 afterward. > > · While the rolling restart is occurring, data on schema > version 1 and schema version 2 is simultaneously being written to the topic. > > · The Kafka connector has to close the current avro file for a > partition and ship it whenever it detects a schema change, which is > happening several times due to the rolling nature of the schema update > deployment and the mixture of message versions being written during this > time. This process causes the overall consumer throughput to plummet. > > Am I reasoning correctly about what we’re observing here? Is there any > way to avoid this when we change schemas (short of stopping all instances > of the service and bringing them up together on the new schema version)? > > Thanks, > Dave > > > >
Re: Kafka connector throughput reduction upon avro schema change
Bumping this. Has anyone here observed this in their Kafka connect deployments? Thanks, Dave On 5/26/17, 1:44 PM, "Dave Hamilton" wrote: We are currently using the Kafka S3 connector to ship Avro data to S3. We made a change to one of our Avro schemas and have noticed consumer throughput on the Kafka connector drop considerably. I am wondering if there is anything we can do to avoid such issues when we update schemas in the future? This is what I believe is happening: · The avro producer application is running on 12 instances. They are restarted in a rolling fashion, switching from producing schema version 1 before the restart to schema version 2 afterward. · While the rolling restart is occurring, data on schema version 1 and schema version 2 is simultaneously being written to the topic. · The Kafka connector has to close the current avro file for a partition and ship it whenever it detects a schema change, which is happening several times due to the rolling nature of the schema update deployment and the mixture of message versions being written during this time. This process causes the overall consumer throughput to plummet. Am I reasoning correctly about what we’re observing here? Is there any way to avoid this when we change schemas (short of stopping all instances of the service and bringing them up together on the new schema version)? Thanks, Dave
Re: Kafka connector throughput reduction upon avro schema change
Hi, does anyone have advice on how to deal with this issue? Is it possible that changing a schema compatibility setting could correct it? Thanks, Dave On 5/26/17, 1:44 PM, "Dave Hamilton" wrote: We are currently using the Kafka S3 connector to ship Avro data to S3. We made a change to one of our Avro schemas and have noticed consumer throughput on the Kafka connector drop considerably. I am wondering if there is anything we can do to avoid such issues when we update schemas in the future? This is what I believe is happening: · The avro producer application is running on 12 instances. They are restarted in a rolling fashion, switching from producing schema version 1 before the restart to schema version 2 afterward. · While the rolling restart is occurring, data on schema version 1 and schema version 2 is simultaneously being written to the topic. · The Kafka connector has to close the current avro file for a partition and ship it whenever it detects a schema change, which is happening several times due to the rolling nature of the schema update deployment and the mixture of message versions being written during this time. This process causes the overall consumer throughput to plummet. Am I reasoning correctly about what we’re observing here? Is there any way to avoid this when we change schemas (short of stopping all instances of the service and bringing them up together on the new schema version)? Thanks, Dave
Kafka connector throughput reduction upon avro schema change
We are currently using the Kafka S3 connector to ship Avro data to S3. We made a change to one of our Avro schemas and have noticed consumer throughput on the Kafka connector drop considerably. I am wondering if there is anything we can do to avoid such issues when we update schemas in the future? This is what I believe is happening: · The avro producer application is running on 12 instances. They are restarted in a rolling fashion, switching from producing schema version 1 before the restart to schema version 2 afterward. · While the rolling restart is occurring, data on schema version 1 and schema version 2 is simultaneously being written to the topic. · The Kafka connector has to close the current avro file for a partition and ship it whenever it detects a schema change, which is happening several times due to the rolling nature of the schema update deployment and the mixture of message versions being written during this time. This process causes the overall consumer throughput to plummet. Am I reasoning correctly about what we’re observing here? Is there any way to avoid this when we change schemas (short of stopping all instances of the service and bringing them up together on the new schema version)? Thanks, Dave
Re: kafka connector for mongodb as a source
There is some log noise in there from Reflections, but it does look like
your connector & task are being created:
[2017-03-27 18:33:00,057] INFO Instantiated task mongodb-0 with version
0.10.0.1 of type org.apache.kafka.connect.mongodb.MongodbSourceTask
(org.apache.kafka.connect.runtime.Worker:264)
And I see the producer configs for the source task's underlying producer
being logged. Then we see the following, suggesting some sort of connection
is being made successfully:
[2017-03-27 18:33:00,397] INFO Source task WorkerSourceTask{id=mongodb-0}
finished initialization and start
(org.apache.kafka.connect.runtime.WorkerSourceTask:138)
[2017-03-27 18:33:00,442] INFO No server chosen by
ReadPreferenceServerSelector{readPreference=primary} from cluster
description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE,
all=[ServerDescription{address=localhost:27017, type=UNKNOWN,
state=CONNECTING}]}. Waiting for 3 ms before timing out
(org.mongodb.driver.cluster:71)
[2017-03-27 18:33:00,455] INFO Opened connection
[connectionId{localValue:1, serverValue:4}] to localhost:27017
(org.mongodb.driver.connection:71)
[2017-03-27 18:33:00,457] INFO Monitor thread successfully connected to
server with description ServerDescription{address=localhost:27017,
type=STANDALONE, state=CONNECTED, ok=true,
version=ServerVersion{versionList=[3, 2, 12]}, minWireVersion=0,
maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=536169}
(org.mongodb.driver.cluster:71)
[2017-03-27 18:33:00,491] INFO Opened connection
[connectionId{localValue:2, serverValue:5}] to localhost:27017
(org.mongodb.driver.connection:71)
But then the logs stop. The framework should just be calling poll() on your
source task. Perhaps you could add some logging to your code to give some
hint as to where it is getting stuck? You could also try increasing the log
level for the framework to DEBUG or even TRACE.
-Ewen
On Mon, Mar 27, 2017 at 6:22 AM, VIVEK KUMAR MISHRA 13BIT0066 <
vivekkumar.mishra2...@vit.ac.in> wrote:
> Hi All,
>
> I am creating kafka connector for mongodb as a source .My connector is
> starting and connecting with kafka but it is not committing any offset.
>
> This is output after starting connector.
>
> [root@localhost kafka_2.11-0.10.1.1]# bin/connect-standalone.sh
> config/connect-standalone.properties config/mongodb.properties
> [2017-03-27 18:32:58,019] INFO StandaloneConfig values:
> rest.advertised.host.name = null
> task.shutdown.graceful.timeout.ms = 5000
> rest.host.name = null
> rest.advertised.port = null
> bootstrap.servers = [localhost:9092]
> offset.flush.timeout.ms = 5000
> offset.flush.interval.ms = 1
> rest.port = 8083
> internal.key.converter = class
> org.apache.kafka.connect.json.JsonConverter
> access.control.allow.methods =
> access.control.allow.origin =
> offset.storage.file.filename = /tmp/connect.offsets
> internal.value.converter = class
> org.apache.kafka.connect.json.JsonConverter
> value.converter = class org.apache.kafka.connect.json.
> JsonConverter
> key.converter = class org.apache.kafka.connect.json.JsonConverter
> (org.apache.kafka.connect.runtime.standalone.StandaloneConfig:178)
> [2017-03-27 18:32:58,162] INFO Logging initialized @609ms
> (org.eclipse.jetty.util.log:186)
> [2017-03-27 18:32:58,392] INFO Kafka Connect starting
> (org.apache.kafka.connect.runtime.Connect:52)
> [2017-03-27 18:32:58,392] INFO Herder starting
> (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:70)
> [2017-03-27 18:32:58,393] INFO Worker starting
> (org.apache.kafka.connect.runtime.Worker:113)
> [2017-03-27 18:32:58,393] INFO Starting FileOffsetBackingStore with file
> /tmp/connect.offsets
> (org.apache.kafka.connect.storage.FileOffsetBackingStore:60)
> [2017-03-27 18:32:58,398] INFO Worker started
> (org.apache.kafka.connect.runtime.Worker:118)
> [2017-03-27 18:32:58,398] INFO Herder started
> (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:72)
> [2017-03-27 18:32:58,398] INFO Starting REST server
> (org.apache.kafka.connect.runtime.rest.RestServer:98)
> [2017-03-27 18:32:58,493] INFO jetty-9.2.15.v20160210
> (org.eclipse.jetty.server.Server:327)
> [2017-03-27 18:32:59,621] INFO HV01: Hibernate Validator 5.1.2.Final
> (org.hibernate.validator.internal.util.Version:27)
> Mar 27, 2017 6:32:59 PM org.glassfish.jersey.internal.Errors logErrors
> WARNING: The following warnings have been detected: WARNING: The
> (sub)resource method listConnectors in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource
> contains
> empty path annotation.
> WARNING: The (sub)resource method createConnector in
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourc
Re: Kafka connector
If you want both N2 and N3 to get all the same messages (rather than each getting an exclusive partitioned subset of the data) then you need to configure N2 and N3 to be in unique Kafka consumer groups which I believe is driven off the "name" of the N2 and N3 connectors. Make sure N2 and N3 have different names. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Thu, Apr 6, 2017 at 4:26 PM, Tushar Sudhakar Jee wrote: > Hello Sir/Ma'am, > I was trying to write a simple case of using kafka connector. My setup > involves using three nodes N1,N2 and N3. > N1 is the source and N2, N3 are the sink nodes in my case. > I am writing data to a text file(say input.txt) on Node N1 and using the > standalone kafka connector I wish to see a text file with content similar > to input.txt on the nodes N2 and N3. > > I am using the REST API to make changes in* topic name, file name and > tasks.max*. > > However, during the experiments I ran I was unable to get a complete copy > of the input.txt on both nodes(N2 and N3) at the *same time. * > *Also tuning the value of tasks.max on nodes (N2 and N3) for the sink > connector decided on which node data would be sent. * > > So, my question is whether I am wrong in expecting such an outcome? > If so then what should I be expecting as a result of the experiment? > If not then how do I get my desired outcome? > > > Regards, > > -- > > *Tushar Sudhakar Jee *| Software Engineer > > c *424.535.8225 <(424)%20535-8225>* | tus...@levyx.com > > [image: Levyx-Logo-Final 9%] > > 49 Discovery, Suite #220 > > Irvine, CA 92618 > > *www.levyx.com <http://www.levyx.com/>* > > > Levyx | 49 Discovery, Suite #220 | Irvine, CA 92618 | www.levyx.com > > This email and any files transmitted with it are confidential and intended > solely for the use of the individual or entity to whom they are addressed. > If you have received this email in error please let us know by e-mail > reply and delete it from your system; copying this message or disclosing > its contents to anyone is strictly prohibited. >
Kafka connector
Hello Sir/Ma'am, I was trying to write a simple case of using kafka connector. My setup involves using three nodes N1,N2 and N3. N1 is the source and N2, N3 are the sink nodes in my case. I am writing data to a text file(say input.txt) on Node N1 and using the standalone kafka connector I wish to see a text file with content similar to input.txt on the nodes N2 and N3. I am using the REST API to make changes in* topic name, file name and tasks.max*. However, during the experiments I ran I was unable to get a complete copy of the input.txt on both nodes(N2 and N3) at the *same time. * *Also tuning the value of tasks.max on nodes (N2 and N3) for the sink connector decided on which node data would be sent. * So, my question is whether I am wrong in expecting such an outcome? If so then what should I be expecting as a result of the experiment? If not then how do I get my desired outcome? Regards, -- *Tushar Sudhakar Jee *| Software Engineer c *424.535.8225* | tus...@levyx.com [image: Levyx-Logo-Final 9%] 49 Discovery, Suite #220 Irvine, CA 92618 *www.levyx.com <http://www.levyx.com/>* -- Levyx | 49 Discovery, Suite #220 | Irvine, CA 92618 | www.levyx.com This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please let us know by e-mail reply and delete it from your system; copying this message or disclosing its contents to anyone is strictly prohibited.
kafka connector for mongodb as a source
Hi All,
I am creating kafka connector for mongodb as a source .My connector is
starting and connecting with kafka but it is not committing any offset.
This is output after starting connector.
[root@localhost kafka_2.11-0.10.1.1]# bin/connect-standalone.sh
config/connect-standalone.properties config/mongodb.properties
[2017-03-27 18:32:58,019] INFO StandaloneConfig values:
rest.advertised.host.name = null
task.shutdown.graceful.timeout.ms = 5000
rest.host.name = null
rest.advertised.port = null
bootstrap.servers = [localhost:9092]
offset.flush.timeout.ms = 5000
offset.flush.interval.ms = 1
rest.port = 8083
internal.key.converter = class
org.apache.kafka.connect.json.JsonConverter
access.control.allow.methods =
access.control.allow.origin =
offset.storage.file.filename = /tmp/connect.offsets
internal.value.converter = class
org.apache.kafka.connect.json.JsonConverter
value.converter = class org.apache.kafka.connect.json.JsonConverter
key.converter = class org.apache.kafka.connect.json.JsonConverter
(org.apache.kafka.connect.runtime.standalone.StandaloneConfig:178)
[2017-03-27 18:32:58,162] INFO Logging initialized @609ms
(org.eclipse.jetty.util.log:186)
[2017-03-27 18:32:58,392] INFO Kafka Connect starting
(org.apache.kafka.connect.runtime.Connect:52)
[2017-03-27 18:32:58,392] INFO Herder starting
(org.apache.kafka.connect.runtime.standalone.StandaloneHerder:70)
[2017-03-27 18:32:58,393] INFO Worker starting
(org.apache.kafka.connect.runtime.Worker:113)
[2017-03-27 18:32:58,393] INFO Starting FileOffsetBackingStore with file
/tmp/connect.offsets
(org.apache.kafka.connect.storage.FileOffsetBackingStore:60)
[2017-03-27 18:32:58,398] INFO Worker started
(org.apache.kafka.connect.runtime.Worker:118)
[2017-03-27 18:32:58,398] INFO Herder started
(org.apache.kafka.connect.runtime.standalone.StandaloneHerder:72)
[2017-03-27 18:32:58,398] INFO Starting REST server
(org.apache.kafka.connect.runtime.rest.RestServer:98)
[2017-03-27 18:32:58,493] INFO jetty-9.2.15.v20160210
(org.eclipse.jetty.server.Server:327)
[2017-03-27 18:32:59,621] INFO HV01: Hibernate Validator 5.1.2.Final
(org.hibernate.validator.internal.util.Version:27)
Mar 27, 2017 6:32:59 PM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: The
(sub)resource method listConnectors in
org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains
empty path annotation.
WARNING: The (sub)resource method createConnector in
org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains
empty path annotation.
WARNING: The (sub)resource method listConnectorPlugins in
org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource
contains empty path annotation.
WARNING: The (sub)resource method serverInfo in
org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty
path annotation.
[2017-03-27 18:33:00,015] INFO Started
o.e.j.s.ServletContextHandler@44e3760b{/,null,AVAILABLE}
(org.eclipse.jetty.server.handler.ContextHandler:744)
[2017-03-27 18:33:00,042] INFO Started ServerConnector@7f58ad44{HTTP/1.1}{
0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:266)
[2017-03-27 18:33:00,043] INFO Started @2492ms
(org.eclipse.jetty.server.Server:379)
[2017-03-27 18:33:00,043] INFO REST server listening at
http://127.0.0.1:8083/, advertising URL http://127.0.0.1:8083/
(org.apache.kafka.connect.runtime.rest.RestServer:150)
[2017-03-27 18:33:00,043] INFO Kafka Connect started
(org.apache.kafka.connect.runtime.Connect:58)
[2017-03-27 18:33:00,048] INFO ConnectorConfig values:
connector.class =
org.apache.kafka.connect.mongodb.MongodbSourceConnector
tasks.max = 1
name = mongodb
value.converter = null
key.converter = null
(org.apache.kafka.connect.runtime.ConnectorConfig:178)
[2017-03-27 18:33:00,048] INFO Creating connector mongodb of type
org.apache.kafka.connect.mongodb.MongodbSourceConnector
(org.apache.kafka.connect.runtime.Worker:159)
[2017-03-27 18:33:00,051] INFO Instantiated connector mongodb with version
0.10.0.1 of type class
org.apache.kafka.connect.mongodb.MongodbSourceConnector
(org.apache.kafka.connect.runtime.Worker:162)
[2017-03-27 18:33:00,053] INFO Finished creating connector mongodb
(org.apache.kafka.connect.runtime.Worker:173)
[2017-03-27 18:33:00,053] INFO SourceConnectorConfig values:
connector.class =
org.apache.kafka.connect.mongodb.MongodbSourceConnector
tasks.max = 1
name = mongodb
value.converter = null
key.converter = null
(org.apache.kafka.connect.runtime.SourceConnectorConfig:178)
[2017-03-27 18:33:00,056] INFO Creating task mongodb-0
(org.apache.kafka.connect.runtime.Worker:252)
[2017-03-27 18:33:00,056] INFO ConnectorConfig values:
connector.class =
org.apache.kafka.connect.mongodb.MongodbSourceConnector
Re: New Kafka Connector
Chris, I think you meant to link to https://github.com/wepay/kafka-connect-bigquery :) - Samuel On Mon, Aug 22, 2016 at 4:24 PM, Chris Egerton wrote: > Hi there, > > We've recently open-sourced a BigQuery sink connector and would like to > request that it be added to the Kafka Connector Hub ( > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Connector+Hub). > The > project can be found at https://github.com/wepay/kafka-connect-biquery, > and > the connector itself has been deployed to Maven Central (latest version is > 0.2.1, but it may still be in the process of synching at the time of > writing). Is there anything else you'd like to know about it before posting > it to your page? > > Cheers! > > Chris Egerton > Software Engineering Intern, WePay >
New Kafka Connector
Hi there, We've recently open-sourced a BigQuery sink connector and would like to request that it be added to the Kafka Connector Hub ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Connector+Hub). The project can be found at https://github.com/wepay/kafka-connect-biquery, and the connector itself has been deployed to Maven Central (latest version is 0.2.1, but it may still be in the process of synching at the time of writing). Is there anything else you'd like to know about it before posting it to your page? Cheers! Chris Egerton Software Engineering Intern, WePay
RE: Running kafka connector application
The twitter connector pom builds a fat jar with all dependencies. You need to add this to the classpath before you start Connect. This is what the Confluent scripts are doing. Regards Andrew From: Ewen Cheslack-Postava<mailto:e...@confluent.io> Sent: 14/06/2016 07:35 To: users@kafka.apache.org<mailto:users@kafka.apache.org> Subject: Re: Running kafka connector application Kanagha, I'm not sure about that particular connector, but normally the build script would provide support for collecting the necessary dependencies. Then all you need to do is add something like /path/to/connector-and-deps/* to your classpath and it shouldn't be affected by versions in the pom.xml, you'd just rebuild to pull in the new dependencies. For example, Confluent's connectors are all setup so if you build with mvn package, it'll include directories with all the dependency jars included. -Ewen On Sun, Jun 12, 2016 at 2:36 PM, Kanagha wrote: > Hi, > > > I'm running the TwitterProducer task as per > https://github.com/Eneco/kafka-connect-twitter > > connect-standalone /connect-source-standalone.properties > /twitter-source.properties > > I see that I have to set the CLASSPATH to include all the dependent jars > that the target connector jar is dependent on. This method wouldn't be > robust if the version changes in pom.xml > > Is there an easier approach to run the standlone connect application? > > > Thanks > Kanagha > > > > > > Kanagha > -- Thanks, Ewen
Re: How to gracefully shutdown Kafka Connector
There's no API for connectors to shut themselves down because that doesn't really fit the streaming model that Kafka Connect works with -- it isn't a batch processing system. If you want to shut down a connector, you'd normally accomplish this via the REST API. Technically you *could* accomplish this by making an HTTP request from your connector, but I'm not sure where or how you'd determine that *none* of your connector's tasks have produced any data for some time. -Ewen On Tue, Jun 14, 2016 at 7:04 AM, Florian Hussonnois wrote: > Hi everyone, > > I would like to know if there is a way to shutdown a connector > programmatically ? > > On my project we have developped a sink-connector to write messages into > GZIP files for testing purposes. We would like to stop the connector after > no message is received for an elapsed time > > Thanks, > > -- > Florian HUSSONNOIS > -- Thanks, Ewen
How to gracefully shutdown Kafka Connector
Hi everyone, I would like to know if there is a way to shutdown a connector programmatically ? On my project we have developped a sink-connector to write messages into GZIP files for testing purposes. We would like to stop the connector after no message is received for an elapsed time Thanks, -- Florian HUSSONNOIS
Re: Starting Kafka Connector via JMX
No. Connectors are started either by passing properties files on the command line (standalone mode) or by submitting connectors to the REST API (in either standalone or distributed mode). -Ewen On Mon, Jun 13, 2016 at 5:42 PM, Abhinav Solan wrote: > Hi Everyone, > > Is there a way to start Kafka Connector via JMX? > > Thanks, > Abhinav > -- Thanks, Ewen
Re: Running kafka connector application
Kanagha, I'm not sure about that particular connector, but normally the build script would provide support for collecting the necessary dependencies. Then all you need to do is add something like /path/to/connector-and-deps/* to your classpath and it shouldn't be affected by versions in the pom.xml, you'd just rebuild to pull in the new dependencies. For example, Confluent's connectors are all setup so if you build with mvn package, it'll include directories with all the dependency jars included. -Ewen On Sun, Jun 12, 2016 at 2:36 PM, Kanagha wrote: > Hi, > > > I'm running the TwitterProducer task as per > https://github.com/Eneco/kafka-connect-twitter > > connect-standalone /connect-source-standalone.properties > /twitter-source.properties > > I see that I have to set the CLASSPATH to include all the dependent jars > that the target connector jar is dependent on. This method wouldn't be > robust if the version changes in pom.xml > > Is there an easier approach to run the standlone connect application? > > > Thanks > Kanagha > > > > > > Kanagha > -- Thanks, Ewen
Starting Kafka Connector via JMX
Hi Everyone, Is there a way to start Kafka Connector via JMX? Thanks, Abhinav
Running kafka connector application
Hi, I'm running the TwitterProducer task as per https://github.com/Eneco/kafka-connect-twitter connect-standalone /connect-source-standalone.properties /twitter-source.properties I see that I have to set the CLASSPATH to include all the dependent jars that the target connector jar is dependent on. This method wouldn't be robust if the version changes in pom.xml Is there an easier approach to run the standlone connect application? Thanks Kanagha Kanagha
Re: Are key.converter.schemas.enable and value.converter.schemas.enable of any use in Kafka connector?
key.converter and value.converter are namespace prefixes in this case. These settings are used by the JsonConverter https://github.com/apache/kafka/blob/trunk/connect/json/src/main/java/org/apache/kafka/connect/json/JsonConverter.java#L53 If schemas are enabled, all JSON messages are sent using an envelope which also carries the schema. If it is disabled, just the usual JSON document will be written, but downstream consumers of the data won't have any information about the schema and will therefore need to handle arbitrary data. Most users will want to keep the schemas disabled since that's the norm for JSON. However, this limits the ability to use a lot of connectors because they need schema information to be able to process the data (e.g. to write to HDFS and load that data into Hive or write to a relational database table we need the schema information so we can setup the target table correctly). -Ewen On Wed, Jun 1, 2016 at 12:41 AM, jingguo yao wrote: > In config/connect-standalone.properties and > config/connect-distributed.properties, there are the following > configuration entries: > > > key.converter.schemas.enable=false > > value.converter.schemas.enable=false > > But there is no Java source code which uses these two configuration > entries. I am talking about kafka_2.11-0.9.0.1. > > > -- > Jingguo > -- Thanks, Ewen
Are key.converter.schemas.enable and value.converter.schemas.enable of any use in Kafka connector?
In config/connect-standalone.properties and config/connect-distributed.properties, there are the following configuration entries: > key.converter.schemas.enable=false > value.converter.schemas.enable=false But there is no Java source code which uses these two configuration entries. I am talking about kafka_2.11-0.9.0.1. -- Jingguo
Re: Kafka-to-Kafka Connector ?
Thnx, that looks like a good option. I'm a bit concerned about running/monitoring an additional external app, an in-stream solution (Connector or Streams plugin) would be preferable. But mirroring may be good enough until we eventually upgrade to 0.10. On Thu, May 5, 2016 at 10:57 AM, tao xiao wrote: > You can use the built-in mirror maker to mirror data from one Kafka to the > other. http://kafka.apache.org/documentation.html#basic_ops_mirror_maker > > On Thu, 5 May 2016 at 10:47 Dean Arnold wrote: > > > I'm developing a Streams plugin for Kafka 0.10, to be run in a dev > sandbox, > > but pull data from a production 0.9 Kafka deployment. Is there a source > > connector that can be used from the 0.10 sandbox to connect to the 0.9 > > cluster ? Given the number of changes/features in 0.10, such a connector > > would be useful for dev/test before upgrading an existing production > > deployment. > > >
Re: Kafka-to-Kafka Connector ?
You can use the built-in mirror maker to mirror data from one Kafka to the other. http://kafka.apache.org/documentation.html#basic_ops_mirror_maker On Thu, 5 May 2016 at 10:47 Dean Arnold wrote: > I'm developing a Streams plugin for Kafka 0.10, to be run in a dev sandbox, > but pull data from a production 0.9 Kafka deployment. Is there a source > connector that can be used from the 0.10 sandbox to connect to the 0.9 > cluster ? Given the number of changes/features in 0.10, such a connector > would be useful for dev/test before upgrading an existing production > deployment. >
Kafka-to-Kafka Connector ?
I'm developing a Streams plugin for Kafka 0.10, to be run in a dev sandbox, but pull data from a production 0.9 Kafka deployment. Is there a source connector that can be used from the 0.10 sandbox to connect to the 0.9 cluster ? Given the number of changes/features in 0.10, such a connector would be useful for dev/test before upgrading an existing production deployment.
Re: Kafka Connector for Solr
Thank you, Surendra. I've added your connector to the Connector Hub page: http://www.confluent.io/developers/connectors On Fri, Apr 22, 2016 at 10:11 PM, Surendra , Manchikanti wrote: > Hi Jay, > > Thanks!! Can you please share the contact person to include this in > Confluent Coneector Hub page. > > Regards, > Surendra M > > -- Surendra Manchikanti > > On Fri, Apr 22, 2016 at 4:32 PM, Jay Kreps wrote: > >> This is great! >> >> -Jay >> >> On Fri, Apr 22, 2016 at 2:28 PM, Surendra , Manchikanti < >> surendra.manchika...@gmail.com> wrote: >> >> > Hi, >> > >> > I have implemented KafkaConnector for Solr, Please find the below github >> > link. >> > >> > https://github.com/msurendra/kafka-connect-solr >> > >> > The initial release having SolrSinkConnector Only, SolrSourceConnector >> > under development will add it soon. >> > >> > Regards, >> > Surendra M >> > >>
Re: Kafka Connector for Solr
Hi Jay, Thanks!! Can you please share the contact person to include this in Confluent Coneector Hub page. Regards, Surendra M -- Surendra Manchikanti On Fri, Apr 22, 2016 at 4:32 PM, Jay Kreps wrote: > This is great! > > -Jay > > On Fri, Apr 22, 2016 at 2:28 PM, Surendra , Manchikanti < > surendra.manchika...@gmail.com> wrote: > > > Hi, > > > > I have implemented KafkaConnector for Solr, Please find the below github > > link. > > > > https://github.com/msurendra/kafka-connect-solr > > > > The initial release having SolrSinkConnector Only, SolrSourceConnector > > under development will add it soon. > > > > Regards, > > Surendra M > > >
Re: Kafka Connector for Solr
This is great! -Jay On Fri, Apr 22, 2016 at 2:28 PM, Surendra , Manchikanti < surendra.manchika...@gmail.com> wrote: > Hi, > > I have implemented KafkaConnector for Solr, Please find the below github > link. > > https://github.com/msurendra/kafka-connect-solr > > The initial release having SolrSinkConnector Only, SolrSourceConnector > under development will add it soon. > > Regards, > Surendra M >
Kafka Connector for Solr
Hi, I have implemented KafkaConnector for Solr, Please find the below github link. https://github.com/msurendra/kafka-connect-solr The initial release having SolrSinkConnector Only, SolrSourceConnector under development will add it soon. Regards, Surendra M
