from:"ORIOL LOPEZ SANCHEZ"

Re: Flink Elasticsearch upsert document in ES

2020-02-10 Thread ORIOL LOPEZ SANCHEZ

When building the request, you should build an UpdateRequest, like the 
following snippet:

import org.elasticsearch.action.update.UpdateRequest
import org.elasticsearch.common.xcontent.XContentType

val doc: String = ???
val targetIndex: String = ???
val indexType: Option[String] = ???

new UpdateRequest()
  .index(targetIndex)
  .`type`(indexType.getOrElse("_doc"))
  .id(id)
  .upsert(doc, XContentType.JSON)
  .doc(doc, XContentType.JSON)
  .docAsUpsert(true)

I'm not entirely sure if you need both "doc" and "upsert" fields, as I think 
this depends on the Elasticsearch you're using.


De: ApoorvK 
Enviat el: dilluns, 10 de febrer de 2020 15:33
Per a: user@flink.apache.org 
Tema: Flink Elasticsearch upsert document in ES

Team,
Presently I have added elasticsearch as a sink to a stream and inserting the
json data, the problem is when I restore the application in case of crash it
reprocess the data in between (meanwhile a backend application updates the
document in ES) and flink reinsert the document in ES and all update to ES
are lost .

I am trying for a update or insert in case document not found or do not
insert if document is already there.


I have tried by providing opType to elasticsearch builder, I am getting an
error message "document already exists" on my console, but it still updates
the value in elasticsearch

val jsonString = write(record)
val rqst: IndexRequest = Requests.indexRequest
  .index(parameter.get("esIndexName"))
  .`type`(parameter.get("esIndexType"))
  .id(record.getApi_key + "_" + record.getOrder_id)
  .source(jsonString, XContentType.JSON)
.opType(OpType.CREATE)



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/



Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede 
contener informaci?n privilegiada o confidencial y es para uso exclusivo de la 
persona o entidad de destino. Si no es usted. el destinatario indicado, queda 
notificado de que la lectura, utilizaci?n, divulgaci?n y/o copia sin 
autorizaci?n puede estar prohibida en virtud de la legislaci?n vigente. Si ha 
recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente 
por esta misma v?a y proceda a su destrucci?n.

The information contained in this transmission is privileged and confidential 
information intended only for the use of the individual or entity named above. 
If the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this transmission in error, do not 
read it. Please immediately reply to the sender that you have received this 
communication in error and then delete it.

Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinat?rio, pode 
conter informa??o privilegiada ou confidencial e ? para uso exclusivo da pessoa 
ou entidade de destino. Se n?o ? vossa senhoria o destinat?rio indicado, fica 
notificado de que a leitura, utiliza??o, divulga??o e/ou c?pia sem autoriza??o 
pode estar proibida em virtude da legisla??o vigente. Se recebeu esta mensagem 
por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e 
proceda a sua destrui??o

Re: Are there plans to support Hadoop 2.9.0 on near future?

2017-11-29 Thread ORIOL LOPEZ SANCHEZ

Thanks, it helped a lot!

But I've seen on
https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+and+Feature+Plan#FlinkReleaseandFeaturePlan-Flink1.4
that they estimated releasing 1.4 at September. Do you know if it will be
released this year or we may have to wait longer?

Thanks a lot.

De: Kostas Kloudas <k.klou...@data-artisans.com>
Enviat el: dimecres, 29 de novembre de 2017 11:15:16
Per a: ORIOL LOPEZ SANCHEZ
A/c: user@flink.apache.org
Tema: Re: Are there plans to support Hadoop 2.9.0 on near future?

Hi Oriol,

As you may have seen form the mailing list we are currently in the process of
releasing Flink 1.4. This is going
to be a hadoop-free distribution which means that it should work with any
hadoop version, including Hadoop 2.9.0.

Given this, I would recommend to try out the release candidate (which will
hopefully be the next official Flink release)
and it should work just fine!

Hope this helps,
Kostas

On Nov 29, 2017, at 10:37 AM, ORIOL LOPEZ SANCHEZ
<oriol.lopezsanc...@telefonica.com<mailto:oriol.lopezsanc...@telefonica.com>>
wrote:

Hi, I'm currently working on designing a data-processing cluster, and one of
the distributed processing tools we want to use is Flink.

As we're creating our cluster from barebones, without relying on any Hadoop
distributions such as Hortonworks or Cloudera, we want to use Flink with Hadoop
2.9.0, but it's not officially supported.

May I know if there are plans to support Hadoop 2.9.0 on a near future?

Thank you very much,

Oriol López Sánchez.

Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede
contener información privilegiada o confidencial y es para uso exclusivo de la
persona o entidad de destino. Si no es usted. el destinatario indicado, queda
notificado de que la lectura, utilización, divulgación y/o copia sin
autorización puede estar prohibida en virtud de la legislación vigente. Si ha
recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente
por esta misma vía y proceda a su destrucción.

The information contained in this transmission is privileged and confidential
information intended only for the use of the individual or entity named above.
If the reader of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this transmission in error, do not
read it. Please immediately reply to the sender that you have received this
communication in error and then delete it.

Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode
conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa
ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica
notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização
pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem
por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e
proceda a sua destruição

Re: Flink Elasticsearch upsert document in ES

Re: Are there plans to support Hadoop 2.9.0 on near future?

2 matches

Site Navigation

Mail list logo

Footer information