[jira] [Created] (FLINK-18372) NullPointerException can happen in SlotPoolImpl#maybeRemapOrphanedAllocation

[Zhu Zhu (Jira)]

Zhu Zhu created FLINK-18372:
---

 Summary: NullPointerException can happen in 
SlotPoolImpl#maybeRemapOrphanedAllocation
 Key: FLINK-18372
 URL: https://issues.apache.org/jira/browse/FLINK-18372
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.12.0
Reporter: Zhu Zhu
 Fix For: 1.12.0


NullPointerException can happen in SlotPoolImpl#maybeRemapOrphanedAllocation, 
which indicates a bug.

https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_apis/build/builds/8189/logs/115

6:07:07,950 [flink-akka.actor.default-dispatcher-7] WARN  
org.apache.flink.runtime.taskexecutor.TaskExecutor   [] - Slot offering 
to JobManager failed. Freeing the slots and returning them to the 
ResourceManager.
java.lang.NullPointerException: null
at 
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.maybeRemapOrphanedAllocation(SlotPoolImpl.java:599)
 ~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at 
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFulfillSlotRequestOrMakeAvailable(SlotPoolImpl.java:564)
 ~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at 
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.offerSlot(SlotPoolImpl.java:701)
 ~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at 
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.offerSlots(SlotPoolImpl.java:625)
 ~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at 
org.apache.flink.runtime.jobmaster.JobMaster.offerSlots(JobMaster.java:541) 
~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_242]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_242]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_242]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_242]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:284)
 ~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:199)
 ~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at 
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
 ~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
 ~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) 
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) 
[akka-actor_2.11-2.5.21.jar:2.5.21]
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) 
[scala-library-2.11.12.jar:?]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) 
[akka-actor_2.11-2.5.21.jar:2.5.21]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) 
[scala-library-2.11.12.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) 
[scala-library-2.11.12.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) 
[scala-library-2.11.12.jar:?]
at akka.actor.Actor$class.aroundReceive(Actor.scala:517) 
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) 
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) 
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.actor.ActorCell.invoke(ActorCell.scala:561) 
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) 
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.dispatch.Mailbox.run(Mailbox.scala:225) 
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235) 
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) 
[akka-actor_2.11-2.5.21.jar:2.5.21]
at 
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) 
[akka-actor_2.11-2.5.21.jar:2.5.21]
at 
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) 
[akka-actor_2.11-2.5.21.jar:2.5.21]
at 
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 
[akka-actor_2.11-2.5.21.jar:2.5.21]
16:07:07,977 [flink-akka.actor.default-dispatcher-7] INFO  
org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl [] - Free slot 
TaskSlot(index:0, state:ACTIVE, resource profile: 
ResourceProfile{cpuCores=0.5000, taskHeapMemory=64.000mb (67108864 
bytes), 

Re: [DISCUSS] Upgrade HBase connector to 2.2.x

[Jark Wu]

+1 to support HBase 2.x
But not sure about dropping support for 1.4.x

I cc'ed to user@ and user-zh@ to hear more feedback from users.

Best,
Jark

On Thu, 18 Jun 2020 at 21:25, Gyula Fóra  wrote:

> Hi All!
>
> I would like to revive an old ticket
>  and discussion around
> upgrading the HBase version of the connector.
>
> The current HBase version is 1.4.3 which is over 2 years old at this point
> and incompatible with the newer HBase versions used at many companies.
>
> We propose to upgrade the connector to the latest version and drop support
> for the old version starting from the 1.12 Flink release. This would help
> us maintain and improve the HBase connector over time.
>
> If the community agrees we are happy to contribute this upgrade as we have
> already developed and tested the updated version.
>
> Cheers,
> Gyula
>

[jira] [Created] (FLINK-18371) NPE of "org.apache.flink.table.data.util.DataFormatConverters$BigDecimalConverter.toExternalImpl(DataFormatConverters.java:680)"

[xiaojin.wy (Jira)]

xiaojin.wy created FLINK-18371:
--

 Summary: NPE of 
"org.apache.flink.table.data.util.DataFormatConverters$BigDecimalConverter.toExternalImpl(DataFormatConverters.java:680)"
 Key: FLINK-18371
 URL: https://issues.apache.org/jira/browse/FLINK-18371
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.11.0
 Environment: I use the sql-gateway to run this sql.
*The sql is:*
CREATE TABLE `src` (
key bigint,
v varchar
) WITH (
'connector'='filesystem',
'csv.field-delimiter'='|',

'path'='/defender_test_data/daily_regression_stream_hive_1.10/test_cast/sources/src.csv',
'csv.null-literal'='',
'format'='csv'
)

select
cast(key as decimal(10,2)) as c1,
cast(key as char(10)) as c2,
cast(key as varchar(10)) as c3
from src
order by c1, c2, c3
limit 1

*The input data is:*
238|val_238
86|val_86
311|val_311
27|val_27
165|val_165
409|val_409
255|val_255
278|val_278
98|val_98
484|val_484
265|val_265
193|val_193
401|val_401
150|val_150
273|val_273
224|val_224
369|val_369
66|val_66
128|val_128
213|val_213
146|val_146
406|val_406
429|val_429
374|val_374
152|val_152
469|val_469
145|val_145
495|val_495
37|val_37
327|val_327
281|val_281
277|val_277
209|val_209
15|val_15
82|val_82
403|val_403
166|val_166
417|val_417
430|val_430
252|val_252
292|val_292
219|val_219
287|val_287
153|val_153
193|val_193
338|val_338
446|val_446
459|val_459
394|val_394
237|val_237
482|val_482
174|val_174
413|val_413
494|val_494
207|val_207
199|val_199
466|val_466
208|val_208
174|val_174
399|val_399
396|val_396
247|val_247
417|val_417
489|val_489
162|val_162
377|val_377
397|val_397
309|val_309
365|val_365
266|val_266
439|val_439
342|val_342
367|val_367
325|val_325
167|val_167
195|val_195
475|val_475
17|val_17
113|val_113
155|val_155
203|val_203
339|val_339
0|val_0
455|val_455
128|val_128
311|val_311
316|val_316
57|val_57
302|val_302
205|val_205
149|val_149
438|val_438
345|val_345
129|val_129
170|val_170
20|val_20
489|val_489
157|val_157
378|val_378
221|val_221
92|val_92
111|val_111
47|val_47
72|val_72
4|val_4
280|val_280
35|val_35
427|val_427
277|val_277
208|val_208
356|val_356
399|val_399
169|val_169
382|val_382
498|val_498
125|val_125
386|val_386
437|val_437
469|val_469
192|val_192
286|val_286
187|val_187
176|val_176
54|val_54
459|val_459
51|val_51
138|val_138
103|val_103
239|val_239
213|val_213
216|val_216
430|val_430
278|val_278
176|val_176
289|val_289
221|val_221
65|val_65
318|val_318
332|val_332
311|val_311
275|val_275
137|val_137
241|val_241
83|val_83
333|val_333
180|val_180
284|val_284
12|val_12
230|val_230
181|val_181
67|val_67
260|val_260
404|val_404
384|val_384
489|val_489
353|val_353
373|val_373
272|val_272
138|val_138
217|val_217
84|val_84
348|val_348
466|val_466
58|val_58
8|val_8
411|val_411
230|val_230
208|val_208
348|val_348
24|val_24
463|val_463
431|val_431
179|val_179
172|val_172
42|val_42
129|val_129
158|val_158
119|val_119
496|val_496
0|val_0
322|val_322
197|val_197
468|val_468
393|val_393
454|val_454
100|val_100
298|val_298
199|val_199
191|val_191
418|val_418
96|val_96
26|val_26
165|val_165
327|val_327
230|val_230
205|val_205
120|val_120
131|val_131
51|val_51
404|val_404
43|val_43
436|val_436
156|val_156
469|val_469
468|val_468
308|val_308
95|val_95
196|val_196
288|val_288
481|val_481
457|val_457
98|val_98
282|val_282
197|val_197
187|val_187
318|val_318
318|val_318
409|val_409
470|val_470
137|val_137
369|val_369
316|val_316
169|val_169
413|val_413
85|val_85
77|val_77
0|val_0
490|val_490
87|val_87
364|val_364
179|val_179
118|val_118
134|val_134
395|val_395
282|val_282
138|val_138
238|val_238
419|val_419
15|val_15
118|val_118
72|val_72
90|val_90
307|val_307
19|val_19
435|val_435
10|val_10
277|val_277
273|val_273
306|val_306
224|val_224
309|val_309
389|val_389
327|val_327
242|val_242
369|val_369
392|val_392
272|val_272
331|val_331
401|val_401
242|val_242
452|val_452
177|val_177
226|val_226
5|val_5
497|val_497
402|val_402
396|val_396
317|val_317
395|val_395
58|val_58
35|val_35
336|val_336
95|val_95
11|val_11
168|val_168
34|val_34
229|val_229
233|val_233
143|val_143
472|val_472
322|val_322
498|val_498
160|val_160
195|val_195
42|val_42
321|val_321
430|val_430
119|val_119
489|val_489
458|val_458
78|val_78
76|val_76
41|val_41
223|val_223
492|val_492
149|val_149
449|val_449
218|val_218
228|val_228
138|val_138
453|val_453
30|val_30
209|val_209
64|val_64
468|val_468
76|val_76
74|val_74
342|val_342
69|val_69
230|val_230
33|val_33
368|val_368
103|val_103
296|val_296
113|val_113
216|val_216
367|val_367
344|val_344
167|val_167
274|val_274
219|val_219
239|val_239
485|val_485
116|val_116
223|val_223
256|val_256
263|val_263
70|val_70
487|val_487
480|val_480
401|val_401
288|val_288
191|val_191
5|val_5
244|val_244
438|val_438
128|val_128
467|val_467
432|val_432

S3 Checkpointing taking long time with stateful operations

[Kathula, Sandeep]

Hi,
We are running a stateful application in Flink with RocksDB as backend and set 
incremental state to true with checkpoints written to S3.

  *   10 task managers each with 2 task slots
  *   Checkpoint interval 3 minutes
  *   Checkpointing mode – At-least once processing

After running app for 2-3 days, we are seeing end to end checkpoint takes 
almost 2 minutes with Sync time 2 sec and async time 15 sec max. But initially 
when state is less, it takes 10-15 sec for checkpointing. As checkpointing mode 
is at least once, align duration is 0. We are seeing a dip in processing during 
this time. Couldn’t find out what the actual issue is.

We also tried with remote HDFS for checkpointing but observed similar behavior.

We have couple of questions:

  *   When sync time is max 2 sec and async time is 15 sec why is end to end 
checkpointing taking almost 2 minutes?
  *   How can we reduce the checkpoint time?
[A screenshot of a cell phone  Description automatically generated]

Any help would be appreciated.


Thank you
Sandeep Kathula

Re: [ANNOUNCE] Apache Flink 1.11.0, release candidate #2

[Robert Metzger]

Thanks a lot for creating another RC.

The good news first: WordCount works on my Mac :)

I didn't find this information in the email: This is the commit the RC is
based on:
https://github.com/apache/flink/commit/c4132de4a50ab9b8f653c69af1ba15af44ff29a2

Additionally, I've created [1] a docker image based on the "
flink-1.11.0-bin-scala_2.12.tgz
"
binary release candidate. It's on DockerHub:
"rmetzger/flink:1.11.0-rc2-c4132de4a50ab9b8f653c69af1ba15af44ff29a2".

Happy testing :)

[1]
https://github.com/rmetzger/flink-docker-factory/runs/785600229?check_suite_focus=true

On Thu, Jun 18, 2020 at 2:41 PM Piotr Nowojski  wrote:

> Hi all,
>
> Apache Flink-1.11.0-RC2 has been created. It has all the artifacts that we
> would typically have for a release.
>
> This RC might have still a couple of missing licensing notices (like the
> one mentioned by Jingsong Li), but as for today morning, there were no open
> release blocking bugs. No official vote will take place for it, but you can
> treat it as a solid base for release testing. It includes the following:
>
>   * The preview source release and binary convenience releases [1], which
> are signed with the key with fingerprint
> 2DA85B93244FDFA19A6244500653C0A2CEA00D0E [2],
>   * All artifacts that would normally be deployed to the Maven Central
> Repository [3]
>
> To test with these artifacts, you can create a settings.xml file with the
> content shown below [4]. This settings file can be referenced in your maven
> commands
> via --settings /path/to/settings.xml. This is useful for creating a
> quickstart project based on the staged release and also for building
> against the staged jars.
>
> Happy testing!
>
> Best,
> Piotrek
>
> [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc2/
> [2] https://dist.apache.org/repos/dist/release/flink/KEYS
> [3]
> https://repository.apache.org/content/repositories/orgapacheflink-1374/
> [4]
> 
>
> flink-1.11.0
>
>
>
>flink-1.11.0
>
>  
>flink-1.11.0
>
> https://repository.apache.org/content/repositories/orgapacheflink-1374/
> 
> 
> 
>   archetype
>   
> https://repository.apache.org/content/repositories/orgapacheflink-1374/
> 
> 
> 
>
>
> 
>
> śr., 17 cze 2020 o 15:29 Piotr Nowojski  napisał(a):
>
> > Hi all,
> >
> > I would like to give an update about the RC2 status. We are now waiting
> > for a green azure build on one final bug fix before creating RC2. This
> bug
> > fix should be merged late afternoon/early evening Berlin time, so RC2
> will
> > be hopefully created tomorrow morning. Until then I would ask to not
> > merge/backport commits to release-1.11 branch, including bug fixes. If
> you
> > have something that's truly essential and should be treated as a release
> > blocker, please reach out to me or Zhijiang.
> >
> > Best,
> > Piotr Nowojski
> >
>

[jira] [Created] (FLINK-18370) Test Flink on Azure-hosted VMs nightly

[Robert Metzger (Jira)]

Robert Metzger created FLINK-18370:
--

 Summary: Test Flink on Azure-hosted VMs nightly
 Key: FLINK-18370
 URL: https://issues.apache.org/jira/browse/FLINK-18370
 Project: Flink
  Issue Type: Improvement
  Components: Build System / Azure Pipelines
Reporter: Robert Metzger
Assignee: Robert Metzger


There are some tests which happen a lot more frequently on the VMs provided by 
Azure (instead of the CI infrastructure hosted in Alibaba Cloud).

Since we have enough CI resources available (at night), we can add a run on the 
Azure machines to get more visibility into those failures, and to increase the 
stability of personal account CI runs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18369) TableEnvironmentITCase.testStatementSetWithSameSinkTableNames failed on azure

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-18369:
--

 Summary: 
TableEnvironmentITCase.testStatementSetWithSameSinkTableNames failed on azure
 Key: FLINK-18369
 URL: https://issues.apache.org/jira/browse/FLINK-18369
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Legacy Planner
Affects Versions: 1.12.0
Reporter: Piotr Nowojski
 Fix For: 1.12.0


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3778=logs=b2f046ab-ae17-5406-acdc-240be7e870e4=93e5ae06-d194-513d-ba8d-150ef6da1d7c

{noformat}
Caused by: java.io.IOException: Output path 
'/tmp/junit6162277837132243514/junit3149129531927352947.tmp' could not be 
initialized. Canceling task...
at 
org.apache.flink.api.common.io.FileOutputFormat.open(FileOutputFormat.java:229)
at 
org.apache.flink.api.java.io.TextOutputFormat.open(TextOutputFormat.java:88)
at 
org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:205)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
at java.lang.Thread.run(Thread.java:748)
{noformat}

{noformat}
[ERROR] Errors: 
[ERROR]   TableEnvironmentITCase.testStatementSetWithSameSinkTableNames:557 » 
Execution ...
[INFO] 
[ERROR] Tests run: 810, Failures: 0, Errors: 1, Skipped: 13
{noformat}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18368) HadoopRecoverableWriterOldHadoopWithNoTruncateSupportTest.createHDFS fails with "Running in secure mode, but config doesn't have a keytab"

[Robert Metzger (Jira)]

Robert Metzger created FLINK-18368:
--

 Summary: 
HadoopRecoverableWriterOldHadoopWithNoTruncateSupportTest.createHDFS fails with 
"Running in secure mode, but config doesn't have a keytab" 
 Key: FLINK-18368
 URL: https://issues.apache.org/jira/browse/FLINK-18368
 Project: Flink
  Issue Type: Bug
  Components: FileSystems, Tests
Affects Versions: 1.12.0
Reporter: Robert Metzger


https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8184=logs=66592496-52df-56bb-d03e-37509e1d9d0f=ae0269db-6796-5583-2e5f-d84757d711aa



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Re-renaming "Flink Master" back to JobManager

[Marta Paes Moreira]

+1

I've found the term Flink Master a tad bit confusing myself, in the past,
as it's not used consistently throughout the documentation (as you mention).

Thanks for following up on this, Aljoscha!

On Wed, Jun 17, 2020 at 5:16 PM Robert Metzger  wrote:

> Thanks a lot for looking into this!
>
> +1 to your proposal
>
> On Wed, Jun 17, 2020 at 10:55 AM David Anderson 
> wrote:
>
> > Aljoscha,
> >
> > I think this is a step in the right direction.
> >
> > In some cases it may be difficult to talk concretely about the
> > differences between different deployment models (e.g., comparing a k8s
> > per-job cluster to a YARN-based session cluster, which is something I
> > typically present during training) without giving names to the internal
> > components. I'm not convinced we can completely avoid mentioning the
> > JobMaster (and Dispatcher and ResourceManagers) in some (rare) contexts
> --
> > but I don't see this as an argument against the proposed change.
> >
> > David
> >
> > On Mon, Jun 15, 2020 at 2:32 PM Konstantin Knauf 
> > wrote:
> >
> > > Hi Aljoscha,
> > >
> > > sounds good to me. Let’s also make sure we don’t refer to the JobMaster
> > as
> > > Jobmanager anywhere then (code, config).
> > >
> > > I am not sure we can avoid mentioning the Flink ResourceManagers in
> user
> > > facing docs completely. For JobMaster and Dispatcher this seems doable.
> > >
> > > Best,
> > >
> > > Konstantin
> > >
> > > On Mon 15. Jun 2020 at 12:56, Aljoscha Krettek 
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > This came to my mind because of the master/slave discussion in [1]
> and
> > > > the larger discussions about inequality/civil rights happening right
> > now
> > > > in the world. I think for this reason alone we should use a name that
> > > > does not include "master".
> > > >
> > > > We could rename it back to JobManager, which was the name mostly used
> > > > before 2019. Since the beginning of Flink, TaskManager was the term
> > used
> > > > for the worker component/node and JobManager was the term used for
> the
> > > > orchestrating component/node.
> > > >
> > > > Currently our glossary [2] defines these terms (paraphrased by me):
> > > >
> > > >   - "Flink Master": it's the orchestrating component that consists of
> > > > resource manager, dispatcher, and JobManager
> > > >
> > > >   - JobManager: it's the thing that manages a single job and runs as
> > > > part of a "Flink Master"
> > > >
> > > >   - TaskManager: it's the worker process
> > > >
> > > > Prior to the introduction of the glossary the definition of
> JobManager
> > > > would have been:
> > > >
> > > >   - It's the orchestrating component that manages execution of jobs
> and
> > > > schedules work on TaskManagers.
> > > >
> > > > Quite some parts in the code and documentation/configuration options
> > > > still use that older meaning of JobManager. Newer parts of the
> > > > documentation use "Flink Master" instead.
> > > >
> > > > I'm proposing to go back to calling the orchestrating component
> > > > JobManager, which would mean that we have to touch up the
> documentation
> > > > to remove mentions of "Flink Master". I'm also proposing not to
> mention
> > > > the internal components such as resource manager and dispatcher in
> the
> > > > glossary because there are transparent to users.
> > > >
> > > > I'm proposing to go back to JobManager instead of an alternative name
> > > > also because switching to yet another name would mean many more
> changes
> > > > to code/documentation/peoples minds.
> > > >
> > > > What do you all think?
> > > >
> > > > Best,
> > > > Aljoscha
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-18209
> > > > [2]
> > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/concepts/glossary.html
> > > >
> > > --
> > >
> > > Konstantin Knauf
> > >
> > > https://twitter.com/snntrable
> > >
> > > https://github.com/knaufk
> > >
> >
>

[DISCUSS] Upgrade HBase connector to 2.2.x

[Gyula Fóra]

Hi All!

I would like to revive an old ticket
 and discussion around
upgrading the HBase version of the connector.

The current HBase version is 1.4.3 which is over 2 years old at this point
and incompatible with the newer HBase versions used at many companies.

We propose to upgrade the connector to the latest version and drop support
for the old version starting from the 1.12 Flink release. This would help
us maintain and improve the HBase connector over time.

If the community agrees we are happy to contribute this upgrade as we have
already developed and tested the updated version.

Cheers,
Gyula

[jira] [Created] (FLINK-18367) Flink HA Mode in Kubernetes. Fencing token not set

[An (Jira)]

An created FLINK-18367:
--

 Summary: Flink HA Mode in Kubernetes. Fencing token not set
 Key: FLINK-18367
 URL: https://issues.apache.org/jira/browse/FLINK-18367
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.10.1
Reporter: An
 Attachments: taskmanager.log

The issue is similar to https://issues.apache.org/jira/browse/FLINK-12382

I'm testing zetcd + session jobs in k8s. Have 2 job managers and 2 
taskmanagers. Everything works fine, but after I delete the pod with the job 
manager leader, task managers not always can register itselves at the new 
leader. The following exception occurs:

´2020-06-18 13:02:43,555 [Thread=flink-akka.actor.default-dispatcher-3] ERROR 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at 
ResourceManager failed due to an error
java.util.concurrent.CompletionException: 
org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token 
not set: Ignoring message RemoteFencedMessage(bcb7d4652fe53a2f8997dc8c87d641a7, 
RemoteRpcInvocation(registerTaskExecutor(TaskExecutorRegistration, Time))) sent 
to akka.tcp://flink@poc-ha-walle-flink-jobmanager:50010/user/resourcemanager 
because the fencing token is null.
 at 
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
 at 
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
 at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
 at 
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
 at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
 at 
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
 ´

Task managers receive notification that leader was changed but seems 
RpcEndpoint can't refresh fence token for some reason 

 

Attached full log from the task manager pod



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18366) Track E2E test durations centrally

[Robert Metzger (Jira)]

Robert Metzger created FLINK-18366:
--

 Summary: Track E2E test durations centrally
 Key: FLINK-18366
 URL: https://issues.apache.org/jira/browse/FLINK-18366
 Project: Flink
  Issue Type: Improvement
  Components: Build System / CI
Reporter: Robert Metzger


Every now and then, our E2E tests start timing out (see FLINK-16795), because 
they hit the currently configured time-limit.
To better understand what the expected E2E time, and potential performance 
regressions, we should track the test execution duration centrally.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Apache Flink 1.11.0, release candidate #2

[Piotr Nowojski]

Hi all,

Apache Flink-1.11.0-RC2 has been created. It has all the artifacts that we
would typically have for a release.

This RC might have still a couple of missing licensing notices (like the
one mentioned by Jingsong Li), but as for today morning, there were no open
release blocking bugs. No official vote will take place for it, but you can
treat it as a solid base for release testing. It includes the following:

  * The preview source release and binary convenience releases [1], which
are signed with the key with fingerprint
2DA85B93244FDFA19A6244500653C0A2CEA00D0E [2],
  * All artifacts that would normally be deployed to the Maven Central
Repository [3]

To test with these artifacts, you can create a settings.xml file with the
content shown below [4]. This settings file can be referenced in your maven
commands
via --settings /path/to/settings.xml. This is useful for creating a
quickstart project based on the staged release and also for building
against the staged jars.

Happy testing!

Best,
Piotrek

[1] https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc2/
[2] https://dist.apache.org/repos/dist/release/flink/KEYS
[3] https://repository.apache.org/content/repositories/orgapacheflink-1374/
[4]

   
flink-1.11.0
   
   
   
   flink-1.11.0
   
 
   flink-1.11.0
   
https://repository.apache.org/content/repositories/orgapacheflink-1374/



  archetype
  
https://repository.apache.org/content/repositories/orgapacheflink-1374/



   
   


śr., 17 cze 2020 o 15:29 Piotr Nowojski  napisał(a):

> Hi all,
>
> I would like to give an update about the RC2 status. We are now waiting
> for a green azure build on one final bug fix before creating RC2. This bug
> fix should be merged late afternoon/early evening Berlin time, so RC2 will
> be hopefully created tomorrow morning. Until then I would ask to not
> merge/backport commits to release-1.11 branch, including bug fixes. If you
> have something that's truly essential and should be treated as a release
> blocker, please reach out to me or Zhijiang.
>
> Best,
> Piotr Nowojski
>

[jira] [Created] (FLINK-18365) The same sql in a batch env and a streaming env has different value.

[xiaojin.wy (Jira)]

xiaojin.wy created FLINK-18365:
--

 Summary: The same sql in a batch env and a streaming env has 
different value.
 Key: FLINK-18365
 URL: https://issues.apache.org/jira/browse/FLINK-18365
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.11.0
 Environment: I use the sql-gateway to run this sql.

*The input table is:*

 CREATE TABLE `scott_dept` (
deptno INT,
dname VARCHAR,
loc VARCHAR
) WITH (
'format.field-delimiter'='|',
'connector.type'='filesystem',
'format.derive-schema'='true',

'connector.path'='/defender_test_data/daily_regression_stream_blink_sql_1.10/test_scalar/sources/scott_dept.csv',
'format.type'='csv'
)

*The input data is:*

10|ACCOUNTING|NEW YORK
20|RESEARCH|DALLAS
30|SALES|CHICAGO
40|OPERATIONS|BOSTON

Reporter: xiaojin.wy
 Fix For: 1.11.0


*The sql is :*
select deptno, (select count(*) from scott_emp where 1 = 0) as x from scott_dept

*The error:*
In a batch environment, the result value is:10|0\n20|0\n30|0\n40|0
In a streaming environment, the result value 
is:10|None\n20|None\n30|None\n40|None



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18364) A streaming sql cause "org.apache.flink.table.api.ValidationException: Type TIMESTAMP(6) of table field 'rowtime' does not match with the physical type TIMESTAMP(3) of t

[xiaojin.wy (Jira)]

xiaojin.wy created FLINK-18364:
--

 Summary: A streaming sql cause 
"org.apache.flink.table.api.ValidationException: Type TIMESTAMP(6) of table 
field 'rowtime' does not match with the physical type TIMESTAMP(3) of the 
'rowtime' field of the TableSink consumed type"
 Key: FLINK-18364
 URL: https://issues.apache.org/jira/browse/FLINK-18364
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.11.0
Reporter: xiaojin.wy
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Yu Li is now part of the Flink PMC

[Ufuk Celebi]

Congrats! :-)

– Ufuk


On Thu, Jun 18, 2020 at 9:47 AM Marta Paes Moreira 
wrote:

> Awesome! Congratulations, Yu!
>
> On Thu, Jun 18, 2020 at 9:16 AM Zhu Zhu  wrote:
>
> > Congratulations!
> >
> > Thanks,
> > Zhu Zhu
> >
> > Guowei Ma  于2020年6月18日周四 上午10:41写道：
> >
> > > Congratulations , Yu!
> > > Best,
> > > Guowei
> > >
> > >
> > > Yang Wang  于2020年6月18日周四 上午10:36写道：
> > >
> > > > Congratulations , Yu!
> > > >
> > > > Best,
> > > > Yang
> > > >
> > > > Piotr Nowojski  于2020年6月17日周三 下午9:21写道：
> > > >
> > > > > Congratulations :)
> > > > >
> > > > > > On 17 Jun 2020, at 14:53, Yun Tang  wrote:
> > > > > >
> > > > > > Congratulations , Yu! well deserved.
> > > > > >
> > > > > > Best
> > > > > > Yun Tang
> > > > > > 
> > > > > > From: Yu Li 
> > > > > > Sent: Wednesday, June 17, 2020 20:03
> > > > > > To: dev 
> > > > > > Subject: Re: [ANNOUNCE] Yu Li is now part of the Flink PMC
> > > > > >
> > > > > > Thanks everyone! Really happy to work in such a great and
> > encouraging
> > > > > > community!
> > > > > >
> > > > > > Best Regards,
> > > > > > Yu
> > > > > >
> > > > > >
> > > > > > On Wed, 17 Jun 2020 at 19:59, Congxian Qiu <
> qcx978132...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >> Congratulations Yu !
> > > > > >> Best,
> > > > > >> Congxian
> > > > > >>
> > > > > >>
> > > > > >> Thomas Weise  于2020年6月17日周三 下午6:23写道：
> > > > > >>
> > > > > >>> Congratulations!
> > > > > >>>
> > > > > >>>
> > > > > >>> On Wed, Jun 17, 2020, 2:59 AM Fabian Hueske  >
> > > > wrote:
> > > > > >>>
> > > > >  Congrats Yu!
> > > > > 
> > > > >  Cheers, Fabian
> > > > > 
> > > > >  Am Mi., 17. Juni 2020 um 10:20 Uhr schrieb Till Rohrmann <
> > > > >  trohrm...@apache.org>:
> > > > > 
> > > > > > Congratulations Yu!
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Wed, Jun 17, 2020 at 7:53 AM Jingsong Li <
> > > > jingsongl...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Congratulations Yu, well deserved!
> > > > > >>
> > > > > >> Best,
> > > > > >> Jingsong
> > > > > >>
> > > > > >> On Wed, Jun 17, 2020 at 1:42 PM Yuan Mei <
> > > yuanmei.w...@gmail.com>
> > > > >  wrote:
> > > > > >>
> > > > > >>> Congrats, Yu!
> > > > > >>>
> > > > > >>> GXGX & well deserved!!
> > > > > >>>
> > > > > >>> Best Regards,
> > > > > >>>
> > > > > >>> Yuan
> > > > > >>>
> > > > > >>> On Wed, Jun 17, 2020 at 9:15 AM jincheng sun <
> > > > >  sunjincheng...@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > >  Hi all,
> > > > > 
> > > > >  On behalf of the Flink PMC, I'm happy to announce that Yu
> Li
> > > is
> > > > > >> now
> > > > >  part of the Apache Flink Project Management Committee
> (PMC).
> > > > > 
> > > > >  Yu Li has been very active on Flink's Statebackend
> > component,
> > > > > >>> working
> > > > > > on
> > > > >  various improvements, for example the RocksDB memory
> > > management
> > > > > >> for
> > > > > > 1.10.
> > > > >  and keeps checking and voting for our releases, and also
> has
> > > > > > successfully
> > > > >  produced two releases(1.10.0&1.10.1) as RM.
> > > > > 
> > > > >  Congratulations & Welcome Yu Li!
> > > > > 
> > > > >  Best,
> > > > >  Jincheng (on behalf of the Flink PMC)
> > > > > 
> > > > > >>>
> > > > > >>
> > > > > >> --
> > > > > >> Best, Jingsong Lee
> > > > > >>
> > > > > >
> > > > > 
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-18363) Add user classloader to context in DeSerializationSchema

[Timo Walther (Jira)]

Timo Walther created FLINK-18363:


 Summary: Add user classloader to context in DeSerializationSchema
 Key: FLINK-18363
 URL: https://issues.apache.org/jira/browse/FLINK-18363
 Project: Flink
  Issue Type: Improvement
  Components: API / Core
Reporter: Timo Walther


Currently, it is only possible to retrieve the metric group in the open() 
method of DeserializationSchema and SerializationSchema. We should also expose 
the user code classloader.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18362) Shipping jdk by shipping archive

[Noah (Jira)]

Noah created FLINK-18362:


 Summary: Shipping jdk by shipping archive
 Key: FLINK-18362
 URL: https://issues.apache.org/jira/browse/FLINK-18362
 Project: Flink
  Issue Type: Wish
Affects Versions: 1.10.1
Reporter: Noah


Hello,
Our team are running flink cluster on YARN, and it works so well 

h4. Functional requirements
Is there any option to ship archive to YARN applications?

h4. Backgrounds
Recently, one of job has been shut down with jdk8 version related issues.
https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6675699

It's easy problem if we could set latest jdk on 
`containerized.taskmanager.env.JAVA_HOME`.
However, cluster administrator said it's difficult to install the latest jdk on 
all cluster machines.
 
So, we planned to run a job on latest jdk that is shipped via shared resources.
There's an option `yarn.ship-directories` but it's quite slow because jdk has 
large number of files.

If Flink supports to ship archive such as `yarn.ship-archive`, we can ship jdk 
archive to remote machines and use shipped jdk location as JAVA_HOME (using 
`yarn.container-start-command-template` )




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18361) Support username and password options for new Elasticsearch connector

[Yangze Guo (Jira)]

Yangze Guo created FLINK-18361:
--

 Summary: Support username and password options for new 
Elasticsearch connector
 Key: FLINK-18361
 URL: https://issues.apache.org/jira/browse/FLINK-18361
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / ElasticSearch
Affects Versions: 1.12.0
Reporter: Yangze Guo
 Fix For: 1.12.0


We support username and password options for ES connector in FLINK-16788. As we 
introduce new ES connector in FLINK-17027, we should also add this support to 
the new ES connector.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Sql-client lack support for new features

[Jark Wu]

+1 for #1
I think this is what we are currently doing, that forward SQL statements to
TableEnv#executeSql, e.g. FLINK-17113, FLINK-18059.
But IMO the SQL CLI specific statements (EXIT, QUIT) should still stay only
in SQL CLI.

Another idea is that, the reviewer/committer should check tests are both
added for SQL CLI and TableEnv if it is a new statement.

Best,
Jark

On Thu, 18 Jun 2020 at 15:40, Rui Li  wrote:

> Thanks Jingsong for bringing up the discussion. Perhaps we can do both #1
> and #2? I like the idea that SQL Client should just forward SQL statements
> to TableEnvironment. IIUC, with TableEnvironment::executeSql in place, most
> statements can be forwarded. Only a very limited set of statements need to
> be handled by SQL Client, like SET and EXIT.
>
> On Thu, Jun 18, 2020 at 3:19 PM Jingsong Li 
> wrote:
>
> > Hi all,
> >
> > I'd like to start a discussion for the new features lacking support of
> > Sql-client.
> >
> > I've seen the new DDL syntax that SQL client lacks support for many
> times.
> > For every new DDL syntax, we need to add support in sql-client. Add
> > a corresponding SqlCommand in sql-client, otherwise this DDL is still
> > not working in sql-client.
> >
> > But it looks like developers always forgot to add support in sql-client.
> > Lots of DDL features just be added in parser and planner, but lack
> > sql-client support, so users will wait for the next release. Just like:
> > https://issues.apache.org/jira/browse/FLINK-7151
> > https://issues.apache.org/jira/browse/FLINK-17198
> > https://issues.apache.org/jira/browse/FLINK-15468
> > https://issues.apache.org/jira/browse/FLINK-15175
> >
> > How to solve this?
> > I think we have two options:
> >
> > 1. Unify the parser in sql-client and TableEnvironment truly. Really make
> > sql-client have all abilities from TableEnvironment. sql-client just
> > forward sql to TableEnvironment. Can it be?
> > 2. A new testing framework mechanism: We can make sql-related tests more
> > "up front", we can move e2e tests (I think we are doing now) and it cases
> > to sql-client oriented. This may require a new testing framework
> mechanism,
> > for example, we can do something like hive sql testing [1] to
> > sql-client oriented. In this way, the testing can cover more horizontal
> and
> > vertical and it is easy to migrate tests from other systems too. And I
> > think, Flink's DDLs are enough stronger to support pure SQLs testing.
> >
> > What do you think?
> >
> > [1]https://github.com/apache/hive/tree/master/ql/src/test/queries
> >
> > Best,
> > Jingsong
> >
>
>
> --
> Best regards!
> Rui Li
>

[jira] [Created] (FLINK-18360) Flink History Server doesn't show correctly table of Completed jobs when there are no archived jobs are in the archive directory

[Jindrich Vimr (Jira)]

Jindrich Vimr created FLINK-18360:
-

 Summary: Flink History Server doesn't show correctly table of 
Completed jobs when there are no archived jobs are in the archive directory
 Key: FLINK-18360
 URL: https://issues.apache.org/jira/browse/FLINK-18360
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Web Frontend
Affects Versions: 1.10.1, 1.10.0, 1.11.0, 1.10.2, 1.12.0, 1.11.1
Reporter: Jindrich Vimr
 Attachments: flink-hs-correct-no-jobs.png, flink-hs-no-data-shown.png

When the directory defined in ${historyserver.archive.fs.dir} is empty (=there 
are no archived completed jobs), the History Server UI fails to show the 
Completed Jobs table.

It should show the empty table with correct column headers and "No data" icon.

This is due to file ${historyserver.web.tmpdir}/jobs/overview.json not being 
created. This file is fetched by the web UI from url `/jobs/overview`.

This file is correctly created and populated by the History Server if the 
directory defined in ${historyserver.archive.fs.dir} contains any job.

The situation when the Completed Jobs table is not populated normally indicates 
that the History Server stars up and processes the jobs in the archive, so the 
user should wait. 
This happened to us few times, as we waited for the HS to finish the archived 
jobs processing just to find out after hours that the HS has in fact nothing to 
show.

 

The fix is simple, by removing `
{code:java}
if (!events.isEmpty()){code}
condition around 
{code:java}
updateJobOverview(webOverviewDir, webDir){code}
in the 
[HistoryServerArchiveFetcher.java|[https://github.com/apache/flink/blob/master/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java#L288]]

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18359) Improve error-log strategy for Elasticsearch sink for large data documentId conflict when using create mode for `insert ignore` semantics

[rinkako (Jira)]

rinkako created FLINK-18359:
---

 Summary: Improve error-log strategy for Elasticsearch sink for 
large data documentId conflict when using create mode for `insert ignore` 
semantics
 Key: FLINK-18359
 URL: https://issues.apache.org/jira/browse/FLINK-18359
 Project: Flink
  Issue Type: Improvement
Reporter: rinkako


The story is: when a flink job for ingesting large number of records from data 
source, processing and indexing to ES with Elasticsearch sink fault, we may 
restart it from a specific data set which contains lots of data which already 
sink into ES.

At this case, a `INSERT IGNORE` semantics is necessary, and we use `public 
IndexRequest create(boolean create)` with `true` args and ignore the 409 
restStatusCode at a customized ActionRequestFailureHandler to make it work.

But, the `BulkProcessorListener` always log a error event before it calls the 
`failureHandler` in its `afterBulk` method, and will produce tons of error log 
for document id conflict, which we already know and handle them in customized 
ActionRequestFailureHandler.

Therefore, it seems that the error log action at the 
ActionRequestFailureHandler (either the default IgnoringFailureHandler or a 
custom handler) is more flexible ?

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Yu Li is now part of the Flink PMC

[Marta Paes Moreira]

Awesome! Congratulations, Yu!

On Thu, Jun 18, 2020 at 9:16 AM Zhu Zhu  wrote:

> Congratulations!
>
> Thanks,
> Zhu Zhu
>
> Guowei Ma  于2020年6月18日周四 上午10:41写道：
>
> > Congratulations , Yu!
> > Best,
> > Guowei
> >
> >
> > Yang Wang  于2020年6月18日周四 上午10:36写道：
> >
> > > Congratulations , Yu!
> > >
> > > Best,
> > > Yang
> > >
> > > Piotr Nowojski  于2020年6月17日周三 下午9:21写道：
> > >
> > > > Congratulations :)
> > > >
> > > > > On 17 Jun 2020, at 14:53, Yun Tang  wrote:
> > > > >
> > > > > Congratulations , Yu! well deserved.
> > > > >
> > > > > Best
> > > > > Yun Tang
> > > > > 
> > > > > From: Yu Li 
> > > > > Sent: Wednesday, June 17, 2020 20:03
> > > > > To: dev 
> > > > > Subject: Re: [ANNOUNCE] Yu Li is now part of the Flink PMC
> > > > >
> > > > > Thanks everyone! Really happy to work in such a great and
> encouraging
> > > > > community!
> > > > >
> > > > > Best Regards,
> > > > > Yu
> > > > >
> > > > >
> > > > > On Wed, 17 Jun 2020 at 19:59, Congxian Qiu  >
> > > > wrote:
> > > > >
> > > > >> Congratulations Yu !
> > > > >> Best,
> > > > >> Congxian
> > > > >>
> > > > >>
> > > > >> Thomas Weise  于2020年6月17日周三 下午6:23写道：
> > > > >>
> > > > >>> Congratulations!
> > > > >>>
> > > > >>>
> > > > >>> On Wed, Jun 17, 2020, 2:59 AM Fabian Hueske 
> > > wrote:
> > > > >>>
> > > >  Congrats Yu!
> > > > 
> > > >  Cheers, Fabian
> > > > 
> > > >  Am Mi., 17. Juni 2020 um 10:20 Uhr schrieb Till Rohrmann <
> > > >  trohrm...@apache.org>:
> > > > 
> > > > > Congratulations Yu!
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Wed, Jun 17, 2020 at 7:53 AM Jingsong Li <
> > > jingsongl...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Congratulations Yu, well deserved!
> > > > >>
> > > > >> Best,
> > > > >> Jingsong
> > > > >>
> > > > >> On Wed, Jun 17, 2020 at 1:42 PM Yuan Mei <
> > yuanmei.w...@gmail.com>
> > > >  wrote:
> > > > >>
> > > > >>> Congrats, Yu!
> > > > >>>
> > > > >>> GXGX & well deserved!!
> > > > >>>
> > > > >>> Best Regards,
> > > > >>>
> > > > >>> Yuan
> > > > >>>
> > > > >>> On Wed, Jun 17, 2020 at 9:15 AM jincheng sun <
> > > >  sunjincheng...@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > >  Hi all,
> > > > 
> > > >  On behalf of the Flink PMC, I'm happy to announce that Yu Li
> > is
> > > > >> now
> > > >  part of the Apache Flink Project Management Committee (PMC).
> > > > 
> > > >  Yu Li has been very active on Flink's Statebackend
> component,
> > > > >>> working
> > > > > on
> > > >  various improvements, for example the RocksDB memory
> > management
> > > > >> for
> > > > > 1.10.
> > > >  and keeps checking and voting for our releases, and also has
> > > > > successfully
> > > >  produced two releases(1.10.0&1.10.1) as RM.
> > > > 
> > > >  Congratulations & Welcome Yu Li!
> > > > 
> > > >  Best,
> > > >  Jincheng (on behalf of the Flink PMC)
> > > > 
> > > > >>>
> > > > >>
> > > > >> --
> > > > >> Best, Jingsong Lee
> > > > >>
> > > > >
> > > > 
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Sql-client lack support for new features

[Rui Li]

Thanks Jingsong for bringing up the discussion. Perhaps we can do both #1
and #2? I like the idea that SQL Client should just forward SQL statements
to TableEnvironment. IIUC, with TableEnvironment::executeSql in place, most
statements can be forwarded. Only a very limited set of statements need to
be handled by SQL Client, like SET and EXIT.

On Thu, Jun 18, 2020 at 3:19 PM Jingsong Li  wrote:

> Hi all,
>
> I'd like to start a discussion for the new features lacking support of
> Sql-client.
>
> I've seen the new DDL syntax that SQL client lacks support for many times.
> For every new DDL syntax, we need to add support in sql-client. Add
> a corresponding SqlCommand in sql-client, otherwise this DDL is still
> not working in sql-client.
>
> But it looks like developers always forgot to add support in sql-client.
> Lots of DDL features just be added in parser and planner, but lack
> sql-client support, so users will wait for the next release. Just like:
> https://issues.apache.org/jira/browse/FLINK-7151
> https://issues.apache.org/jira/browse/FLINK-17198
> https://issues.apache.org/jira/browse/FLINK-15468
> https://issues.apache.org/jira/browse/FLINK-15175
>
> How to solve this?
> I think we have two options:
>
> 1. Unify the parser in sql-client and TableEnvironment truly. Really make
> sql-client have all abilities from TableEnvironment. sql-client just
> forward sql to TableEnvironment. Can it be?
> 2. A new testing framework mechanism: We can make sql-related tests more
> "up front", we can move e2e tests (I think we are doing now) and it cases
> to sql-client oriented. This may require a new testing framework mechanism,
> for example, we can do something like hive sql testing [1] to
> sql-client oriented. In this way, the testing can cover more horizontal and
> vertical and it is easy to migrate tests from other systems too. And I
> think, Flink's DDLs are enough stronger to support pure SQLs testing.
>
> What do you think?
>
> [1]https://github.com/apache/hive/tree/master/ql/src/test/queries
>
> Best,
> Jingsong
>


-- 
Best regards!
Rui Li

[jira] [Created] (FLINK-18358) TableEnvironmentITCase.testSqlUpdateAndToDataSetWithTableSource:245 unstable: result mismatch

[Robert Metzger (Jira)]

Robert Metzger created FLINK-18358:
--

 Summary:  
TableEnvironmentITCase.testSqlUpdateAndToDataSetWithTableSource:245 unstable: 
result mismatch
 Key: FLINK-18358
 URL: https://issues.apache.org/jira/browse/FLINK-18358
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Affects Versions: 1.12.0
Reporter: Robert Metzger


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3749=logs=b2f046ab-ae17-5406-acdc-240be7e870e4=93e5ae06-d194-513d-ba8d-150ef6da1d7c

{code}
[ERROR]   TableEnvironmentITCase.testSqlUpdateAndToDataSetWithTableSource:245 
expected:<8,24.95375[]
> but was:<8,24.95375[03]
>

{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Sql-client lack support for new features

[Benchao Li]

Thanks Jingsong for bringing up this discussion!

+1 for #1
I think it would always be a good practice to have a unify implementation
for two different user-facing API
of the same functionality.

The divergence of two different implementations always bothers users. In
the past, users complains a lot
about that some features only exist in sql-client but not in Table API.
And for now, we are facing it's counter-part problem, that we added
functionality in Table API, but forgot to
sync them to sql-client.

I'm not very sure whether we can unify all the features of sql-client and
TableEnvironment, maybe some
features only exists in one place. But for the common features, I think it
would be great to have a unify
implementation.

Jingsong Li  于2020年6月18日周四 下午3:19写道：

> Hi all,
>
> I'd like to start a discussion for the new features lacking support of
> Sql-client.
>
> I've seen the new DDL syntax that SQL client lacks support for many times.
> For every new DDL syntax, we need to add support in sql-client. Add
> a corresponding SqlCommand in sql-client, otherwise this DDL is still
> not working in sql-client.
>
> But it looks like developers always forgot to add support in sql-client.
> Lots of DDL features just be added in parser and planner, but lack
> sql-client support, so users will wait for the next release. Just like:
> https://issues.apache.org/jira/browse/FLINK-7151
> https://issues.apache.org/jira/browse/FLINK-17198
> https://issues.apache.org/jira/browse/FLINK-15468
> https://issues.apache.org/jira/browse/FLINK-15175
>
> How to solve this?
> I think we have two options:
>
> 1. Unify the parser in sql-client and TableEnvironment truly. Really make
> sql-client have all abilities from TableEnvironment. sql-client just
> forward sql to TableEnvironment. Can it be?
> 2. A new testing framework mechanism: We can make sql-related tests more
> "up front", we can move e2e tests (I think we are doing now) and it cases
> to sql-client oriented. This may require a new testing framework mechanism,
> for example, we can do something like hive sql testing [1] to
> sql-client oriented. In this way, the testing can cover more horizontal and
> vertical and it is easy to migrate tests from other systems too. And I
> think, Flink's DDLs are enough stronger to support pure SQLs testing.
>
> What do you think?
>
> [1]https://github.com/apache/hive/tree/master/ql/src/test/queries
>
> Best,
> Jingsong
>


-- 

Best,
Benchao Li

[DISCUSS] Sql-client lack support for new features

[Jingsong Li]

Hi all,

I'd like to start a discussion for the new features lacking support of
Sql-client.

I've seen the new DDL syntax that SQL client lacks support for many times.
For every new DDL syntax, we need to add support in sql-client. Add
a corresponding SqlCommand in sql-client, otherwise this DDL is still
not working in sql-client.

But it looks like developers always forgot to add support in sql-client.
Lots of DDL features just be added in parser and planner, but lack
sql-client support, so users will wait for the next release. Just like:
https://issues.apache.org/jira/browse/FLINK-7151
https://issues.apache.org/jira/browse/FLINK-17198
https://issues.apache.org/jira/browse/FLINK-15468
https://issues.apache.org/jira/browse/FLINK-15175

How to solve this?
I think we have two options:

1. Unify the parser in sql-client and TableEnvironment truly. Really make
sql-client have all abilities from TableEnvironment. sql-client just
forward sql to TableEnvironment. Can it be?
2. A new testing framework mechanism: We can make sql-related tests more
"up front", we can move e2e tests (I think we are doing now) and it cases
to sql-client oriented. This may require a new testing framework mechanism,
for example, we can do something like hive sql testing [1] to
sql-client oriented. In this way, the testing can cover more horizontal and
vertical and it is easy to migrate tests from other systems too. And I
think, Flink's DDLs are enough stronger to support pure SQLs testing.

What do you think?

[1]https://github.com/apache/hive/tree/master/ql/src/test/queries

Best,
Jingsong

Re: [ANNOUNCE] Yu Li is now part of the Flink PMC

[Zhu Zhu]

Congratulations!

Thanks,
Zhu Zhu

Guowei Ma  于2020年6月18日周四 上午10:41写道：

> Congratulations , Yu!
> Best,
> Guowei
>
>
> Yang Wang  于2020年6月18日周四 上午10:36写道：
>
> > Congratulations , Yu!
> >
> > Best,
> > Yang
> >
> > Piotr Nowojski  于2020年6月17日周三 下午9:21写道：
> >
> > > Congratulations :)
> > >
> > > > On 17 Jun 2020, at 14:53, Yun Tang  wrote:
> > > >
> > > > Congratulations , Yu! well deserved.
> > > >
> > > > Best
> > > > Yun Tang
> > > > 
> > > > From: Yu Li 
> > > > Sent: Wednesday, June 17, 2020 20:03
> > > > To: dev 
> > > > Subject: Re: [ANNOUNCE] Yu Li is now part of the Flink PMC
> > > >
> > > > Thanks everyone! Really happy to work in such a great and encouraging
> > > > community!
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Wed, 17 Jun 2020 at 19:59, Congxian Qiu 
> > > wrote:
> > > >
> > > >> Congratulations Yu !
> > > >> Best,
> > > >> Congxian
> > > >>
> > > >>
> > > >> Thomas Weise  于2020年6月17日周三 下午6:23写道：
> > > >>
> > > >>> Congratulations!
> > > >>>
> > > >>>
> > > >>> On Wed, Jun 17, 2020, 2:59 AM Fabian Hueske 
> > wrote:
> > > >>>
> > >  Congrats Yu!
> > > 
> > >  Cheers, Fabian
> > > 
> > >  Am Mi., 17. Juni 2020 um 10:20 Uhr schrieb Till Rohrmann <
> > >  trohrm...@apache.org>:
> > > 
> > > > Congratulations Yu!
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Wed, Jun 17, 2020 at 7:53 AM Jingsong Li <
> > jingsongl...@gmail.com>
> > > > wrote:
> > > >
> > > >> Congratulations Yu, well deserved!
> > > >>
> > > >> Best,
> > > >> Jingsong
> > > >>
> > > >> On Wed, Jun 17, 2020 at 1:42 PM Yuan Mei <
> yuanmei.w...@gmail.com>
> > >  wrote:
> > > >>
> > > >>> Congrats, Yu!
> > > >>>
> > > >>> GXGX & well deserved!!
> > > >>>
> > > >>> Best Regards,
> > > >>>
> > > >>> Yuan
> > > >>>
> > > >>> On Wed, Jun 17, 2020 at 9:15 AM jincheng sun <
> > >  sunjincheng...@gmail.com>
> > > >>> wrote:
> > > >>>
> > >  Hi all,
> > > 
> > >  On behalf of the Flink PMC, I'm happy to announce that Yu Li
> is
> > > >> now
> > >  part of the Apache Flink Project Management Committee (PMC).
> > > 
> > >  Yu Li has been very active on Flink's Statebackend component,
> > > >>> working
> > > > on
> > >  various improvements, for example the RocksDB memory
> management
> > > >> for
> > > > 1.10.
> > >  and keeps checking and voting for our releases, and also has
> > > > successfully
> > >  produced two releases(1.10.0&1.10.1) as RM.
> > > 
> > >  Congratulations & Welcome Yu Li!
> > > 
> > >  Best,
> > >  Jincheng (on behalf of the Flink PMC)
> > > 
> > > >>>
> > > >>
> > > >> --
> > > >> Best, Jingsong Lee
> > > >>
> > > >
> > > 
> > > >>>
> > > >>
> > >
> > >
> >
>

[jira] [Created] (FLINK-18357) ContinuousFileReaderOperator checkpoint timeout when files and directories get larger

[Jark Wu (Jira)]

Jark Wu created FLINK-18357:
---

 Summary: ContinuousFileReaderOperator checkpoint timeout when 
files and directories get larger
 Key: FLINK-18357
 URL: https://issues.apache.org/jira/browse/FLINK-18357
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
Reporter: Jark Wu


This is reported from user-zh, I tranlsated it into this issue:

{{env.readFile(format,path, FileProcessingMode.PROCESS_CONTINUOUSLY, 6)}}

It monitors a directory of A, we will generate a new directory under A every 
day, such as:

A/20200101/
A/20200102/
A/20200103/
...
...

However, as time goes on, it has to monitor 200 directories and 500 files in 
each directory until June. Then everytime, the checkpoint has to synchronize 
offsets of 200*500 files which is very large and checkpoint gets timeout. 



My though is that is it possible to have a configuration, to mark a 
sub-directory as idle after a inactive interval, then we don't need to 
checkpoint all the file offsets under it. 




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18356) Exit code 137 returned from process when testing pyflink

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-18356:
--

 Summary: Exit code 137 returned from process when testing pyflink
 Key: FLINK-18356
 URL: https://issues.apache.org/jira/browse/FLINK-18356
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.12.0
Reporter: Piotr Nowojski


{preformated}
= test session starts ==
platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
cachedir: .tox/py37-cython/.pytest_cache
rootdir: /__w/3/s/flink-python
collected 568 items

pyflink/common/tests/test_configuration.py ..[  1%]
pyflink/common/tests/test_execution_config.py ...[  5%]
pyflink/dataset/tests/test_execution_environment.py .
##[error]Exit code 137 returned from process: file name '/bin/docker', 
arguments 'exec -i -u 1002 
97fc4e22522d2ced1f4d23096b8929045d083dd0a99a4233a8b20d0489e9bddb 
/__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
Finishing: Test - python

{preformated}

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3729=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3



--
This message was sent by Atlassian Jira
(v8.3.4#803005)