Error parsing annotations in flink-table-planner_blink_2.12-1.11.2

2020-11-01 Thread Yuval Itzchakov
this issue? -- Best Regards, Yuval Itzchakov.

Connecting the new DataSource API (FLIP-27) with DynamicTableSource

2020-11-01 Thread Yuval Itzchakov
rns a ScanRuntimeProvider interface, which only has an implementation for the old SourceFunction[T], and not for the new Source[T] class: [image: image.png] Does the Table API not yet support the new Source API? or am I missing something? -- Best Regards, Yuval Itzchakov.

Re: Error parsing annotations in flink-table-planner_blink_2.12-1.11.2

2020-11-02 Thread Yuval Itzchakov
uot;org.apache.flink"%% "flink-streaming-java" % flinkVersion % "test" classifier "tests", "org.apache.flink"%% "flink-test-utils" % flinkVersion % "test", ) On Mon, Nov 2, 2020 at 3:21 PM

Re: Error parsing annotations in flink-table-planner_blink_2.12-1.11.2

2020-11-05 Thread Yuval Itzchakov
. > > Regards, > Timo > > > On 02.11.20 16:00, Aljoscha Krettek wrote: > > But you're not using apiguardian yourself or have it as a dependency > > before this, right? > > > > Best, > > Aljoscha > > > > On 02.11.20 14:59, Yuval Itzch

DataStream.connect semantics for Flink SQL / Table API

2020-11-12 Thread Yuval Itzchakov
I don't want to turn the Table into a DataStream, since I want to leverage predicate pushdown for the definition of the result table. Does anything like this currently exist? -- Best Regards, Yuval Itzchakov.

Re: DataStream.connect semantics for Flink SQL / Table API

2020-11-12 Thread Yuval Itzchakov
/UNION ALL should work because then there won't > be any buffering by event time which could delay your output. > > Have you tried this and have you seen an actual delay in your output? > > Best, > Aljoscha > > On 12.11.20 12:57, Yuval Itzchakov wrote: > > Hi, >

NullPointerException while calling TableEnvironment.sqlQuery in Flink 1.12

2020-12-21 Thread Yuval Itzchakov
ml) -- Best Regards, Yuval Itzchakov.

Registering RawType with LegacyTypeInfo via Flink CREATE TABLE DDL

2020-12-24 Thread Yuval Itzchakov
">" ... "MULTISET" ... "ARRAY" ... "." ... Which looks like the planner doesn't like the round brackets on the LEGACY type. What is the correct way to register the table with this type with Flink? -- Best Regards, Yuval Itzchakov.

Re: Registering RawType with LegacyTypeInfo via Flink CREATE TABLE DDL

2020-12-24 Thread Yuval Itzchakov
PM Yuval Itzchakov wrote: > Hi, > > I have a UDF which returns a type of MAP 'ANY')>. When I try to register this type with Flink via the > CREATE TABLE DDL, I encounter an exception: > > - SQL parse failed. Encountered "(" at line 2, column 256. > Was e

Re: Registering RawType with LegacyTypeInfo via Flink CREATE TABLE DDL

2020-12-27 Thread Yuval Itzchakov
LegacyInfo cannot convert a plain RAW type > back to TypeInformation. > > Did you try to construct type information by a new > fresh TypeInformationRawType ? > > Yuval Itzchakov 于2020年12月24日周四 下午7:24写道: > >> An expansion to my question: >> >> What I really want

Re: Registering RawType with LegacyTypeInfo via Flink CREATE TABLE DDL

2020-12-28 Thread Yuval Itzchakov
ersions.fromLegacyInfoToDataType`. > > Another work around could be that you simply use `BYTES` as the return > type and pass around a byte array instead. > > Maybe you can show us a little end-to-end example, what you are trying > to achieve? > > Regards, > Timo > >

Re: Registering RawType with LegacyTypeInfo via Flink CREATE TABLE DDL

2020-12-28 Thread Yuval Itzchakov
.., LEGACY). Is there any way I can preserve the underlying type and still register the column somehow with CREATE TABLE? On Mon, Dec 28, 2020 at 11:22 AM Yuval Itzchakov wrote: > Hi Timo, > > Thanks for the explanation. Passing around a byte array is not possible > since I need to know t

Re: Registering RawType with LegacyTypeInfo via Flink CREATE TABLE DDL

2020-12-28 Thread Yuval Itzchakov
d > representation of a `CREATE TABLE` statement. In this case you would > have the full control over the entire type stack end-to-end. > > Regards, > Timo > > On 28.12.20 10:36, Yuval Itzchakov wrote: > > Timo, an additional question. > > > > I am currently using Type

Re: Registering RawType with LegacyTypeInfo via Flink CREATE TABLE DDL

2020-12-28 Thread Yuval Itzchakov
. DataType is a super type of TypeInformation. > > Registering types (e.g. giving a RAW type backed by io.circe.Json a name > such that it can be used in a DDL statement) is also in the backlog but > requires another FLIP. > > Regards, > Timo > > > On 28.12.20 13:04, Y

Flink 1.12 Kryo Serialization Error

2021-01-11 Thread Yuval Itzchakov
(err)=> throw err case Right(value) => value } } Would appreciate any help on how to debug this further. -- Best Regards, Yuval Itzchakov.

ProjectWatermarkAssignerTransposeRule field pruning causes wrong watermark index to be accessed

2021-01-25 Thread Yuval Itzchakov
]) Is this the desired behavior? I was quite surprised by this and the fact that the pruning happens regardless of if the underlying table is a TableSource or if it actually filters out these unused fields from the result query. -- Best Regards, Yuval Itzchakov.

Annotating AggregateFunction accumulator type with @DataTypeHint

2021-01-26 Thread Yuval Itzchakov
type? All the examples in the docs refer to Scalar functions. -- Best Regards, Yuval Itzchakov.

Re: HybridMemorySegment index out of bounds exception

2021-01-28 Thread Yuval Itzchakov
nail down the problem. > > Have you tried to replace the JSON object with a regular String? If the > exception is gone after this change. I believe it must be the > serialization and not the network stack. > > Regards, > Timo > > > On 28.01.21 10:29, Yuval Itzchakov wrote:

Re: HybridMemorySegment index out of bounds exception

2021-01-28 Thread Yuval Itzchakov
he.org/jira/browse/FLINK-20986 ? > > Regards, > Timo > > > On 28.01.21 16:30, Yuval Itzchakov wrote: > > Hi Timo, > > > > I tried replacing it with an ordinary ARRAY DataType, which > > doesn't reproduce the issue. > > If I use a RawType

Max with retract aggregate function does not support type: ''CHAR''.

2021-02-02 Thread Yuval Itzchakov
ting the max function: [image: image.png] It does seem like all primitives are supported. Is there a particular reason why a CHAR would not be supported? Is this an oversight? -- Best Regards, Yuval Itzchakov.

Re: Max with retract aggregate function does not support type: ''CHAR''.

2021-02-03 Thread Yuval Itzchakov
: > Thanks for reaching out to the Flink community Yuval. I am pulling in Timo > and Jark who might be able to answer this question. From what I can tell, > it looks a bit like an oversight because VARCHAR is also supported. > > Cheers, > Till > > On Tue, Feb 2, 2021 at 6:12 PM

DynamicTableSink's equivalent of UpsertStreamTableSink.setKeyFields

2021-02-03 Thread Yuval Itzchakov
ay to receive the unique keys for upsert queries with the new DynamicTableSink API? -- Best Regards, Yuval Itzchakov.

Re: DynamicTableSink's equivalent of UpsertStreamTableSink.setKeyFields

2021-02-03 Thread Yuval Itzchakov
e with just a primary key. > > Regards, > Timo > > > On 03.02.21 14:09, Yuval Itzchakov wrote: > > Hi, > > I'm reworking an existing UpsertStreamTableSink into the new > > DynamicTableSink API. In the previous API, one would get the unique keys > > for

Generated SinkConversion code ignores incoming StreamRecord timestamp

2021-02-14 Thread Yuval Itzchakov
Am I missing anything in the Table -> DataStream[Row] conversion that should make the timestamp follow through? or is this a bug? -- Best Regards, Yuval Itzchakov.

Re: Generated SinkConversion code ignores incoming StreamRecord timestamp

2021-02-14 Thread Yuval Itzchakov
s.apache.org/jira/browse/FLINK-21013 > On 15/02/2021 08:20, Yuval Itzchakov wrote: > > Hi, > > I have a source that generates events with timestamps. These flow nicely, > until encountering a conversion from Table -> DataStream[Row]: > > def toRowRetractSt

Re: Role of Rowtime Field Task?

2021-02-20 Thread Yuval Itzchakov
> > that has an upstream Kafka source task. What exactly does the rowtime task > do? > > -- > Thank you, > Aeden > -- Best Regards, Yuval Itzchakov.

Re: Community chat?

2021-02-22 Thread Yuval Itzchakov
A dedicated Slack would be awesome. On Mon, Feb 22, 2021, 22:57 Sebastián Magrí wrote: > Is there any chat from the community? > > I saw the freenode channel but it's pretty dead. > > A lot of the time a more chat alike venue where to discuss stuff > synchronously or just share ideas turns out v

Re: Community chat?

2021-02-24 Thread Yuval Itzchakov
uld Slack be your first pick? Or would something async but > easier to interact with also work, like a Discourse forum? > > Thanks for bringing this up! > > Marta > > > > On Mon, Feb 22, 2021 at 10:03 PM Yuval Itzchakov > wrote: > >> A dedicated Slack would be aweso

Re: Scaling Higher than 10k Nodes

2021-03-04 Thread Yuval Itzchakov
>>> Best, >>> Piotrek >>> >>> pon., 1 mar 2021 o 14:43 Joey Tran >>> napisał(a): >>> >>>> Hi, I was looking at Apache Beam/Flink for some of our data processing >>>> needs, but when reading about the resource managers >>>> (YARN/mesos/Kubernetes), it seems like they all top out at around 10k >>>> nodes. What are recommended solutions for scaling higher than this? >>>> >>>> Thanks in advance, >>>> Joey >>>> >>> -- Best Regards, Yuval Itzchakov.

Re: LocalWatermarkAssigner causes predicate pushdown to be skipped

2021-03-05 Thread Yuval Itzchakov
till use rowtime without > implementing `SupportsWatermarkPushDown` in your custom source. > > I will lookp in Shengkai who worked on this topic recently. > > Regards, > Timo > > > On 04.03.21 18:52, Yuval Itzchakov wrote: > > Bumping this up again, would apprecia

Re: LocalWatermarkAssigner causes predicate pushdown to be skipped

2021-03-07 Thread Yuval Itzchakov
markAssigner if you > assigned a watermark. > If you implement SupportsWatermarkPushdown, the LogicalWatermarkAssigner > will be pushed > into TableSource, and then you can push Filter into source if source > implement SupportsFilterPushdown. > > Best, > Jark > > On Sat, 6 M

Re: LocalWatermarkAssigner causes predicate pushdown to be skipped

2021-03-08 Thread Yuval Itzchakov
mark > assigner or extend the filter push down rule to capture the structure that > the watermark assigner is the parent of the table scan. > > Best, > Shengkai > > Yuval Itzchakov 于2021年3月8日周一 上午12:13写道: > >> Hi Jark, >> >> Even after implementing bo

Re: [DISCUSS] Feature freeze date for 1.13

2021-04-01 Thread Yuval Itzchakov
Hi All, I would really love to merge https://github.com/apache/flink/pull/15307 prior to 1.13 release cutoff, it just needs some more tests which I can hopefully get to today / tomorrow morning. This is a critical fix as now predicate pushdown won't work for any stream which generates a watermark

Re: [DISCUSS] Feature freeze date for 1.13

2021-04-06 Thread Yuval Itzchakov
> Best, > Guowei > > > On Thu, Apr 1, 2021 at 10:33 PM Yuval Itzchakov wrote: > >> Hi All, >> >> I would really love to merge https://github.com/apache/flink/pull/15307 >> prior to 1.13 release cutoff, it just needs some more tests which I can >> h

Re: Flink 1.11.4?

2021-04-13 Thread Yuval Itzchakov
Roman, is there an ETA on 1.13? On Mon, Apr 12, 2021, 16:17 Roman Khachatryan wrote: > Hi Maciek, > > There are no specific plans for 1.11.4 yet as far as I know. > The official policy is to support the current and previous minor > release [1]. So 1.12 and 1.13 will be officially supported once

Blink Planner fails to generate RowtimeAttribute for custom TableSource

2020-05-11 Thread Yuval Itzchakov
d was wondering if anyone else ran into this? -- Best Regards, Yuval Itzchakov.

Infer if a Table will create an AppendStream / RetractStream

2020-05-18 Thread Yuval Itzchakov
preserve these semantics. -- Best Regards, Yuval Itzchakov.

Re: Infer if a Table will create an AppendStream / RetractStream

2020-05-19 Thread Yuval Itzchakov
this information. Which planner and > version are you using? > > Regards, > Timo > > > On 18.05.20 14:16, Yuval Itzchakov wrote: > > Hi, > > > > Is there any way to infer if a Table is going to generate an > > AppendStream or a RetractStream under the hoo

SourceReaderBase not part of flink-core 1.11.1

2020-07-24 Thread Yuval Itzchakov
Hi, I'm implementing a custom SourceReader and want to base it on SourceReaderBase. However, it seems like while SourceReader and Source are part of `flink-core:1.11.1`, SourceReaderBase is not? Do I need an external package for it? -- Best Regards, Yuval Itzchakov.

Table API Kafka Connector Sink Key Definition

2020-08-19 Thread Yuval Itzchakov
ks like it creates a KeyedSerializationSchemaWrapper which just sets the key to null? Would love some help. -- Best Regards, Yuval Itzchakov.

Re: Table API Kafka Connector Sink Key Definition

2020-08-20 Thread Yuval Itzchakov
> If you want to use a Kafka producer which writes the records out with a > key, then please take a look at KafkaSerializationSchema. It supports this > functionality. > > Cheers, > Till > > On Wed, Aug 19, 2020 at 6:36 PM Yuval Itzchakov wrote: > >> Hi, >>

Loading FlinkKafkaProducer fails with LinkError

2020-08-23 Thread Yuval Itzchakov
in.( OperatorChain.java:144) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask .java:370) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) at java.lang.Thread.run(Thread.java:748) -- Best Regards, Yuval Itzchakov.

Re: Loading FlinkKafkaProducer fails with LinkError

2020-08-25 Thread Yuval Itzchakov
u'd not bundle any > Flink dependencies in your uber-jar (use provided scope for them). > > If you have everything bundled in one uber-jar and execute it somehow > without flink-dist, then I don't immediately see a solution. Then the log > with the classpath would help. >

Re: Loading FlinkKafkaProducer fails with LinkError

2020-08-25 Thread Yuval Itzchakov
one uber-jar and execute it somehow >> without flink-dist, then I don't immediately see a solution. Then the log >> with the classpath would help. >> >> Best, >> >> Arvid >> >> >> On Sun, Aug 23, 2020 at 1:37 PM Yuval Itzchakov >>

Re: Loading FlinkKafkaProducer fails with LinkError

2020-08-25 Thread Yuval Itzchakov
arent-first > classloading reinforces the suspicion that they are present in the > distribution. > > As Arvid mentioned, the classpath log entry (at the very start of the log > file) would be interesting. > > Did you build the Flink yourself distribution, or are you relying on one

Re: Loading FlinkKafkaProducer fails with LinkError

2020-08-25 Thread Yuval Itzchakov
OK, I think I figured it out. It looks like the uber-jar is also being placed under `lib`, which is probably the cause of the problem. Question is, why does it identify it as two different versions? It's exactly the same JAR. On Tue, Aug 25, 2020 at 10:22 PM Yuval Itzchakov wrote: > I&

Re: Loading FlinkKafkaProducer fails with LinkError

2020-08-25 Thread Yuval Itzchakov
it is a jar > of an older version of your setup? > > Can you give some details on the NoSuchMethodException? Specifically > whether it tries to access something from the Kafka connector, or from your > own user code. > > On 25/08/2020 21:27, Yuval Itzchakov wrote: > > OK,

Ignoring invalid values in KafkaSerializationSchema

2020-09-23 Thread Yuval Itzchakov
? -- Best Regards, Yuval Itzchakov.

Re: Ignoring invalid values in KafkaSerializationSchema

2020-09-24 Thread Yuval Itzchakov
tting the data to Kafka. >> >> Best, >> Matthias >> >> [1] >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148645988 >> [2] https://issues.apache.org/jira/browse/FLINK-19397 >> >> On Wed, Sep 23, 2020 at 1:12 PM Yuval Itzchak

akka.framesize configuration does not runtime execution

2020-10-15 Thread Yuval Itzchakov
is being recovered from the stored savepoint, thus not allowing me to configure re-configure the maximum size of the payload? -- Best Regards, Yuval Itzchakov.

Re: serialization issue in streaming job run with scala Future

2019-09-19 Thread Yuval Itzchakov
Debshish, could you share an example of before and after of your classes for future reference? On Thu, 19 Sep 2019, 10:42 Debasish Ghosh, wrote: > We solved the problem of serialization by making some things transient > which were being captured as part of the closure. So we no longer have > ser

Re: Challenges Deploying Flink With Savepoints On Kubernetes

2019-09-24 Thread Yuval Itzchakov
t; *Altitude 2019 in San Francisco | Sept. 23 - 25* > It’s not just an IT conference, it’s “a complete learning and networking > experience” > <https://altitude.bettercloud.com/?utm_source=gmail&utm_medium=signature&utm_campaign=2019-altitude> > > -- Best Regards, Yuval Itzchakov.

Re: Challenges Deploying Flink With Savepoints On Kubernetes

2019-09-24 Thread Yuval Itzchakov
om checkpoint after a bug fix. > From what I know, currently you can use checkpoint as a savepoint as well > > Hao Sun > > > On Tue, Sep 24, 2019 at 7:48 AM Yuval Itzchakov wrote: > >> AFAIK there's currently nothing implemented to solve this problem, but >&

Re: How can I get the backpressure signals inside my function or operator?

2019-11-07 Thread Yuval Itzchakov
Hi, We've been dealing with a similar problem of downstream consumers causing backpressure. One idea that a colleague of mine suggested is measuring the time it takes to call Collector[T].out. Since this method is used to push the records downstream, it will also actively block in case the buffer

Re: Streaming data to Segment

2019-11-20 Thread Yuval Itzchakov
Hi Li, You're in the right direction. One additional step would be to use RickSinkFunction[Data] instead of SinkFunction[Data] which exposes open and close functions which allow you to initialize and dispose resources properly. On Thu, 21 Nov 2019, 5:23 Li Peng, wrote: > Hey folks, I'm interest

Does Flink SQL support stop with savepoint draining?

2019-12-31 Thread Yuval Itzchakov
none of them transitioned although none of them were in a state of backpressure at all. I waited around 10 minutes, yet nothing seemed to change. My question is, do Flink SQL streams support draining the streams before stopping? -- Best Regards, Yuval Itzchakov.

Re: Giving useful names to the SQL steps/operators.

2020-02-29 Thread Yuval Itzchakov
Unfortunately, it isn't possible. You can't set names to steps like ordinary Java/Scala functions. On Sat, 29 Feb 2020, 17:11 Niels Basjes, wrote: > Hi, > > I'm playing around with the streaming SQL engine in combination with the > UDF I wrote ( https://yauaa.basjes.nl/UDF-ApacheFlinkTable.html

Restoring state from an incremental RocksDB checkpoint

2020-03-13 Thread Yuval Itzchakov
Hi, We're using RocksDB as a state backend. We've come to a situation where due to high backpressure in one of our operators, we can't make a savepoint complete. Since we have retained previous checkpoints, I was wondering if these would be eligible to serve as a restoration point, given that we

Passing checkpoint lock object to StreamSourceContexts.getSourceContext after StreamTask.getCheckpointLock deprecation

2020-04-09 Thread Yuval Itzchakov
uired checkpointLock? -- Best Regards, Yuval Itzchakov.

Re: Passing checkpoint lock object to StreamSourceContexts.getSourceContext after StreamTask.getCheckpointLock deprecation

2020-04-09 Thread Yuval Itzchakov
One thing that had just occurred to me is that the context is later used to emit watermarks and send elements downstream using ctx.collect. Perhaps all these operations should now be switched to the new mailbox executor instead? On Thu, Apr 9, 2020 at 8:52 PM Yuval Itzchakov wrote: > Hi, &g

Assigning UID to Flink SQL queries

2019-08-27 Thread Yuval Itzchakov
names (essentially the entire query plan) shown in the UI / thread names while debugging. Is there any convenient way to set / override the UID for SQL defined queries? -- Best Regards, Yuval Itzchakov.

Re: Assigning UID to Flink SQL queries

2019-08-30 Thread Yuval Itzchakov
Anyone? On Tue, 27 Aug 2019, 17:23 Yuval Itzchakov, wrote: > Hi, > > We a have a bunch of Flink SQL queries running in our Flink environment. > For > regular Table API interactions, we can override `uid` which also gives us > an > indicative name for the thread/UI to l

Error While Building Flink From Source

2019-09-10 Thread Yuval Itzchakov
l: class Overrides [ERROR] location: class org.apache.flink.streaming.api.operators.StreamSink Has anyone run into this problem? -- Best Regards, Yuval Itzchakov.

Re: Error While Building Flink From Source

2019-09-10 Thread Yuval Itzchakov
t; regards. > > On Tue, Sep 10, 2019 at 9:06 PM Yuval Itzchakov wrote: > >> Hi, >> I'm trying to build Flink from source. I'm using Maven 3.6.1 and >> executing the following command: >> >> mvn clean install -DskipTests -Dfast -Dscala-2.12 >>

Re: Error While Building Flink From Source

2019-09-10 Thread Yuval Itzchakov
Never mind, turns out it was an error on my part. Somehow I managed do add an "S" to an attribute mistakenly :) On Tue, Sep 10, 2019 at 7:29 PM Yuval Itzchakov wrote: > Still getting the same error message using your command. Which Maven > version are you using? > > On Tue

Compound Keys Using Temporal Tables

2019-09-13 Thread Yuval Itzchakov
Hi, Given table X with an event time, A, B and C columns, is there a way to pass a compound key, i.e. A and B as the primaryKey argument of Table.createTemporalFunction? My attempts so far yield a runtime exception where the String doesn't match a given regex.

Re: Multiple select queries in single job on flink table API

2021-04-16 Thread Yuval Itzchakov
Yes. Instead of calling execute on each table, create a StatementSet using your StreamTableEnvironment (tableEnv.createStatementSet) and use addInsert and finally .execute when you want to run the job. On Sat, Apr 17, 2021, 03:20 tbud wrote: > If I want to run two different select queries on

Re: Flink in k8s operators list

2021-05-28 Thread Yuval Itzchakov
https://github.com/lyft/flinkk8soperator On Fri, May 28, 2021, 10:09 Ilya Karpov wrote: > Hi there, > > I’m making a little research about the easiest way to deploy link job to > k8s cluster and manage its lifecycle by *k8s operator*. The list of > solutions is below: > - https://github.com/fint

Alternatives to TypeConversions fromDataTypeToLegacyInfo / fromLegacyInfoToDataType

2021-06-04 Thread Yuval Itzchakov
needs to be converted into a DataStream[Row] and in turn I need to apply some stateful transformations on it. In order to do that I need the TypeInformation[Row] produced in order to pass into the various state functions. @Timo Walther I would love your help on this. -- Best Regards, Yuval Itzchakov.

Re: Alternatives to TypeConversions fromDataTypeToLegacyInfo / fromLegacyInfoToDataType

2021-06-04 Thread Yuval Itzchakov
the near future. > But for now we went with the most generic solution that supports > everything that can come out of Table API. > > Regards, > Timo > > On 04.06.21 15:12, Yuval Itzchakov wrote: > > When upgrading to Flink 1.13, I ran into deprecation warnings on > >

Re: Alternatives to TypeConversions fromDataTypeToLegacyInfo / fromLegacyInfoToDataType

2021-06-04 Thread Yuval Itzchakov
t to TypeInformation. > > Regards, > Timo > > > On 04.06.21 16:05, Yuval Itzchakov wrote: > > Hi Timo, > > Thank you for the response. > > > > The tables being created in reality are based on arbitrary SQL code such > > that I don't know what th

Allow setting job name when using StatementSet

2021-06-07 Thread Yuval Itzchakov
Hi, Currently when using StatementSet, the name of the job is autogenerated by the runtime: [image: image.png] Is there any reason why there shouldn't be an overload that allows the user to explicitly specify the name of the running job? -- Best Regards, Yuval Itzchakov.

Re: Allow setting job name when using StatementSet

2021-06-08 Thread Yuval Itzchakov
e you want >> to have? >> >> Best wishes, >> Nico >> >> On Mon, Jun 7, 2021 at 10:09 AM Yuval Itzchakov >> wrote: >> >>> Hi, >>> >>> Currently when using StatementSet, the name of the job is autogenerated >>> by the

Re: State migration for sql job

2021-06-09 Thread Yuval Itzchakov
As my company is also a heavy user of Flink SQL, the state migration story is very important to us. I as well believe that adding new fields should start to accumulate state from the point in time of the change forward. Is anyone actively working on this? Is there anyway to get involved? On Tue,

Re: RocksDB MapState debugging key serialization

2021-06-30 Thread Yuval Itzchakov
an I find out the serialized values that are being used for key > comparison? Can you recommend any possible solutions or debugging > strategies that would help? > > > > Thank you, > > Tom > -- Best Regards, Yuval Itzchakov.

Flink 1.13 fails to load log4j2 yaml configuration file via jackson-dataformat-yaml

2021-07-04 Thread Yuval Itzchakov
ERROR StatusLogger No logging configuration This indicates that for some reason, the jackson dataformat YAML library is not getting properly loaded from my uber JAR at runtime. Has anyone run into this? Any possible workarounds? -- Best Regards, Yuval Itzchakov.

Flink 1.13 fails to load log4j2 yaml configuration file via jackson-dataformat-yaml

2021-07-05 Thread Yuval Itzchakov
Hi, I am attempting to upgrade Flink from 1.9 to 1.13.1 I am using a YAML based log4j file. In 1.9, it worked perfectly fine by adding the following dependency to the classpath (I deploy with an uber JAR): "com.fasterxml.jackson.dataformat" % "jackson-dataformat-yaml" % "2.12.3" However, with Fl

Re: Flink 1.13 fails to load log4j2 yaml configuration file via jackson-dataformat-yaml

2021-07-07 Thread Yuval Itzchakov
that with improved classloader separation, you actually need > to put your dependency into `lib/` instead of putting it into your user > jar. But I'm pulling in @Chesnay Schepler who has > much more insights. > > On Sun, Jul 4, 2021 at 9:45 PM Yuval Itzchakov wrote: > &

Issue with Flink SQL using RocksDB backend

2021-07-26 Thread Yuval Itzchakov
sk.invoke(StreamTask.java:620) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) at java.base/java.lang.Thread.run(Thread.java:829) Would appreciate help in the direction of how to debug this issue, or if anyone has encountered this before. -- Best Regards, Yuval Itzchakov.

Re: Issue with Flink SQL using RocksDB backend

2021-07-26 Thread Yuval Itzchakov
JING ZHANG wrote: > Hi Yuval, > I run a similar SQL (without `FIRST` aggregate function), there is nothing > wrong. > `FIRST` is a custom aggregate function? Would you please check if there is > a drawback in `FIRST`? Whether the query could run without `FIRST`? > > Best, >

Re: [External] naming table stages

2021-07-28 Thread Yuval Itzchakov
Hi Jing, An additional challenge with the current Table API / SQL approach for iperator naming is that it makes it very hard to export metrics, i.e. to track watermarks with Prometheus, when operator names are not assignable by the user. On Wed, Jul 28, 2021, 13:11 JING ZHANG wrote: > Hi Clemen

Re: Flink 1.13 fails to load log4j2 yaml configuration file via jackson-dataformat-yaml

2021-07-29 Thread Yuval Itzchakov
I'm using the "classloader.resolve-order" = "parent-first" flag due to my lib having some deps that otherwise cause runtime collision errors in Kafka. On Wed, Jul 7, 2021 at 12:25 PM Yuval Itzchakov wrote: > Interesting, I don't have bind explicitly on th

Re: Flink 1.13 fails to load log4j2 yaml configuration file via jackson-dataformat-yaml

2021-07-29 Thread Yuval Itzchakov
tory on the > classpath. > What I don't understand is why it would pick up your log4j file. It should > only use the file that is embedded within BashJavaUtils.jar. > > On 29/07/2021 13:11, Yuval Itzchakov wrote: > > Hi Chesnay, > > So I looked into it, and jackson-d

Re: Flink 1.13 fails to load log4j2 yaml configuration file via jackson-dataformat-yaml

2021-07-29 Thread Yuval Itzchakov
the utils classpath? On Thu, Jul 29, 2021, 17:55 Chesnay Schepler wrote: > That could be...could you try configuring "env.java.opts: > -Dlog4j.configurationFile=..."? > > On 29/07/2021 15:18, Yuval Itzchakov wrote: > > Perhaps because I'm passing

Re: Flink 1.13 fails to load log4j2 yaml configuration file via jackson-dataformat-yaml

2021-07-30 Thread Yuval Itzchakov
e BashJavaUtils. > > On 30/07/2021 08:48, Yuval Itzchakov wrote: > > It is finding the file though, the problem is that the lib/ might not be > on the classpath when the file is being parsed, thus the YAML file is not > recognized as being parsable. > > Is there a wa

Output[StreamRecord[T]] thread safety

2021-08-02 Thread Yuval Itzchakov
Data) in1.getString(0)); } if (isNull$57) { outWriter.setNullAt(0); } else { outWriter.writeString(0, field$57); }outWriter.complete(); return out; } } -- Best Regards, Yuval Itzchakov.

Re: Output[StreamRecord[T]] thread safety

2021-08-03 Thread Yuval Itzchakov
to > writing the results out) in a separated thread. Flink's runtime expect the > whole operator chain to run in a single thread. > > Yuval Itzchakov 于2021年8月3日周二 下午1:47写道: > >> Hi, >> >> Flink 1.13.1 >> Scala 2.12.4 >> >> I have an implementati

Re: [ANNOUNCE] RocksDB Version Upgrade and Performance

2021-08-04 Thread Yuval Itzchakov
d need to trace it back to RocksDB, then one of the > committers can find the relevant patch from RocksDB master and backport it. > That isn't the greatest user experience. > > Because of those disadvantages, we would prefer to do the upgrade to the > newer RocksDB version despite the unfortunate performance regression. > > Best, > Stephan > > > -- Best Regards, Yuval Itzchakov.

Re: Custom Source with the new Data Source API

2021-08-04 Thread Yuval Itzchakov
sed unbound source (i.e. Kafka). > > Thanks! > Bin > -- Best Regards, Yuval Itzchakov.

Re: [ANNOUNCE] RocksDB Version Upgrade and Performance

2021-08-04 Thread Yuval Itzchakov
//github.com/facebook/rocksdb/issues/5774 > [11] http://david-grs.github.io/tls_performance_overhead_cost_linux/ > [12] https://github.com/ververica/frocksdb/pull/19 > [13] https://github.com/facebook/rocksdb/pull/5441/ > [14] https://github.com/facebook/rocksdb/pull/2283 > > >

Understanding the semantics of SourceContext.collect

2021-08-04 Thread Yuval Itzchakov
tate and can rely on that fact that it has synchronously been pushed end to end. -- Best Regards, Yuval Itzchakov.

Re: Production Grade GitOps Based Kubernetes Setup

2021-08-06 Thread Yuval Itzchakov
Hi Niklas, We are currently using the Lyft operator for Flink in production ( https://github.com/lyft/flinkk8soperator), which is additional alternative. The project itself is pretty much in Zombie state, but commits happen every now and then. 1. Native Kubernetes could definitely work with GitOp

UNION ALL on two LookupTableSources

2021-08-15 Thread Yuval Itzchakov
32E10 io, 0.0 network, 0.0 memory}, id = 170 I do understand that the semantics of unioning two lookups may be a bit complicated, but was wondering if this is planned to be supported in the future? -- Best Regards, Yuval Itzchakov.

Validating Flink SQL without registering with StreamTableEnvironment

2021-08-18 Thread Yuval Itzchakov
Ideally, I don't really care about the overall registration process of sources, transformations and sinks, I just want to make sure the syntax is correct from Flinks perspective. Is there any straightforward way of doing this? -- Best Regards, Yuval Itzchakov.

Re: Validating Flink SQL without registering with StreamTableEnvironment

2021-08-18 Thread Yuval Itzchakov
gt; this works for you? > > > Best > Ingo > > On Wed, Aug 18, 2021 at 12:51 PM Yuval Itzchakov > wrote: > >> Hi, >> >> I have a use-case where I need to validate hundreds of Flink SQL queries. >> Ideally, I'd like to run these validations in paralle

Creating a generic ARRAY_AGG aggregate function for Flink SQL

2021-08-23 Thread Yuval Itzchakov
on how to tackle this. -- Best Regards, Yuval Itzchakov.

Re: Creating a generic ARRAY_AGG aggregate function for Flink SQL

2021-08-24 Thread Yuval Itzchakov
;> >> You can also open a JIRA ticket to require a built-in support for >> array_agg, as this function exists in many data ware houses. >> >> Yuval Itzchakov 于2021年8月23日周一 下午7:38写道: >> >>> Hi, >>> >>> I'm trying to implement a generic AR

Job crashing with RowSerializer EOF exception

2021-09-09 Thread Yuval Itzchakov
his or try to get more info would be great. Thanks. -- Best Regards, Yuval Itzchakov.

Re: Job crashing with RowSerializer EOF exception

2021-09-09 Thread Yuval Itzchakov
e > serializers in the serializer chain is somehow broken. > What data type are you serializating? Does it include some type serializer > by a custom serializer, or Kryo, ... ? > > On Thu, Sep 9, 2021 at 4:35 PM Yuval Itzchakov wrote: > >> Hi, >> >> Flink 1.13

Re: Building a flink connector

2021-09-17 Thread Yuval Itzchakov
Hi Lars, We've built a custom connector for Snowflake (both source and sink). Feel free to reach out in private if you have any questions. On Fri, Sep 17, 2021, 14:33 Lars Skjærven wrote: > We're in need of a Google Bigtable flink connector. Do you have any tips > on how this could be done, e.

  1   2   >