[jira] [Created] (FLINK-14948) Implement shutDownCluster for MiniClusterClient

[Zili Chen (Jira)]

Zili Chen created FLINK-14948:
-

 Summary: Implement shutDownCluster for MiniClusterClient
 Key: FLINK-14948
 URL: https://issues.apache.org/jira/browse/FLINK-14948
 Project: Flink
  Issue Type: Improvement
  Components: Client / Job Submission
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14949) Task cancellation can be stuck against out-of-thread error

[Hwanju Kim (Jira)]

Hwanju Kim created FLINK-14949:
--

 Summary: Task cancellation can be stuck against out-of-thread error
 Key: FLINK-14949
 URL: https://issues.apache.org/jira/browse/FLINK-14949
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.8.2
Reporter: Hwanju Kim


Task cancellation 
([_cancelOrFailAndCancelInvokable_|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java#L991])
 relies on multiple separate threads, which are _TaskCanceler_, 
_TaskInterrupter_, and _TaskCancelerWatchdog_. While TaskCanceler performs 
cancellation itself, TaskInterrupter periodically interrupts a non-reacting 
task and TaskCancelerWatchdog kills JVM if cancellation has never been finished 
within a certain amount of time (by default 3 min). Those all ensure that 
cancellation can be done or either aborted transitioning to a terminal state in 
finite time (FLINK-4715).

However, if any asynchronous thread creation is failed such as by out-of-thread 
(_java.lang.OutOfMemoryError: unable to create new native thread_), the code 
transitions to CANCELING, but nothing could be performed for cancellation or 
watched by watchdog. Currently, jobmanager does [retry 
cancellation|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java#L1121]
 against any error returned, but a next retry [returns success once it sees 
CANCELING|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java#L997],
 assuming that it is in progress. This leads to complete stuck in CANCELING, 
which is non-terminal, so state machine is stuck after that.

One solution would be that if a task has transitioned to CANCELLING but it gets 
fatal error or OOM (i.e., _isJvmFatalOrOutOfMemoryError_ is true) indicating 
that it could not reach spawning TaskCancelerWatchdog, it could immediately 
consider that as fatal error (not safely cancellable) calling 
_notifyFatalError_, just as TaskCancelerWatchdog does but eagerly and 
synchronously. That way, it can at least transition out of the non-terminal 
state and furthermore clear potentially leaked thread/memory by restarting JVM. 
The same method is also invoked by _failExternally_, but transitioning to 
FAILED seems less critical as it's already terminal state.

How to reproduce is straightforward by running an application that keeps 
creating threads, each of which never finishes in a loop, and has multiple 
tasks so that one task triggers failure and then the others are attempted to be 
cancelled by full fail-over. In web UI dashboard, some tasks from a task 
manager where any of cancellation-related threads failed to be spawned are 
stuck in CANCELLING for good.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14947) Implement LocalExecutor as new Executor interface

[Zili Chen (Jira)]

Zili Chen created FLINK-14947:
-

 Summary: Implement LocalExecutor as new Executor interface
 Key: FLINK-14947
 URL: https://issues.apache.org/jira/browse/FLINK-14947
 Project: Flink
  Issue Type: Sub-task
  Components: Client / Job Submission
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0


We can replace {{PlanExecutor}} things with new Executor interface. One of this 
series is implement a {{LocalExecutor}} that execute pipeline within a 
{{MiniCluster}}. For correctly lifecycle management I would wait for 
FLINK-14762 being merged.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14946) Retraction infer would result in bad plan under corner case in blink planner

[Jing Zhang (Jira)]

Jing Zhang created FLINK-14946:
--

 Summary: Retraction infer would result in bad plan under corner 
case in blink planner
 Key: FLINK-14946
 URL: https://issues.apache.org/jira/browse/FLINK-14946
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.9.1, 1.9.0
Reporter: Jing Zhang
 Attachments: image-2019-11-26-14-54-34-797.png

Retractions rule would result in bad plan under some case, I simplify the case 
like the following sql, complete test case could be found in attachments.

{code:scala}
  val join_sql =
  """
|SELECT
|  ll.a AS a,
|  ll.b AS b,
|  cnt
|FROM (
| SELECT a, b, COUNT(c) AS cnt FROM l GROUP BY a, b
|) ll
|JOIN (
| SELECT a, b FROM r GROUP BY a, b
|) rr ON
|(ll.a = rr.a AND ll.b = rr.b)
  """.stripMargin !image-2019-11-26-14-52-52-824.png! 

val sqlQuery =
  s"""
 |SELECT a, b_1, SUM(cnt) AS cnt
 |FROM (
 | SELECT *, b AS b_1 FROM (${join_sql})
 |   UNION ALL
 | SELECT *, 'SEA' AS b_1 FROM (${join_sql})
 |) AS total_result
 |GROUP BY a, b_1
  """.stripMargin
{code}

The plan is :
 !image-2019-11-26-14-54-34-797.png! 
After retraction infer, we expect two join node in the above plan has 
`AccRetract` asAccMode. However, AccMode of Join1 is right, accMode of Join2 is 
unexpected.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-86: Improve Connector Properties

[Jark Wu]

Thanks all for the voting. I'm closing the vote now.
So far, the vote received:

* +1 votes (5 binding, 3 non-binding):
 - Dawid (binding)
 - Terry Wang
 - Jark Wu (binding)
 - Jingsong Li
 - Aljoscha (binding)
 - Kurt Young (binding)
 - Xuefu
 - Timo (binding)

* 0/-1 votes: none

Thereby, the community accepted FLIP-6

Cheers,
Jark

On Sat, 23 Nov 2019 at 14:39, Timo Walther  wrote:

> +1 (binding)
>
> Thanks,
> Timo
>
> On 23.11.19 00:40, Xuefu Z wrote:
> > +1 (non-binding)
> >
> > On Wed, Nov 20, 2019 at 4:40 AM Kurt Young  wrote:
> >
> >> +1 (binding)
> >>
> >> Best,
> >> Kurt
> >>
> >>
> >> On Wed, Nov 20, 2019 at 6:56 PM Aljoscha Krettek 
> >> wrote:
> >>
> >>> +1 (binding)
> >>>
> >>> Best,
> >>> Aljoscha
> >>>
>  On 20. Nov 2019, at 11:36, Jingsong Li 
> wrote:
> 
>  +1 (non-binding)
>  Thanks Jark for driving this.
> 
>  Best,
>  Jingsong Lee
> 
>  On Wed, Nov 20, 2019 at 6:25 PM Jark Wu  wrote:
> 
> > +1 (binding)
> >
> > Best,
> > Jark
> >
> > On Wed, 20 Nov 2019 at 18:11, Terry Wang  wrote:
> >
> >> +1 (non-binding)
> >>
> >> Best,
> >> Terry Wang
> >>
> >>
> >>
> >>> 2019年11月20日 17:47，Dawid Wysakowicz  写道：
> >>>
> >>> +1 from my side
> >>>
> >>> Best,
> >>>
> >>> Dawid
> >>>
> >>> On 20/11/2019 10:36, Jark Wu wrote:
>  Hi everyone,
> 
>  I would like to start a vote on FLIP-86. The discussion seems to
> >> have
>  reached an agreement.
> 
>  Please vote for the following design document:
> 
> 
> >>
> >
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-86%3A+Improve+Connector+Properties
> 
>  The discussion can be found at:
> 
> 
> >>
> >
> >>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-86-Improve-Connector-Properties-td34922.html
> 
>  This voting will be open for at least 72 hours.
> 
>  Best,
>  Jark
> 
> >>>
> >>
> >>
> >
> 
> 
>  --
>  Best, Jingsong Lee
> >>>
> >>>
> >>
> >
> >
>
>

[jira] [Created] (FLINK-14945) Simplify some code in runtime

[clay (Jira)]

clay created FLINK-14945:


 Summary: Simplify some code in runtime
 Key: FLINK-14945
 URL: https://issues.apache.org/jira/browse/FLINK-14945
 Project: Flink
  Issue Type: Task
  Components: Runtime / Checkpointing, Runtime / Metrics, Runtime / Task
Reporter: clay
 Fix For: 1.10.0


I found some code to simplify when using IDEA to inspect the code,  so I submit 
a PR here, This change only involves code simplification 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] PyFlink User-Defined Function Resource Management

[Dian Fu]

Thanks for your votes and feedbacks. I have discussed with @Zhu Zhu offline and 
also on the design doc.

It seems that we have reached consensus on the design. I would bring up the 
VOTE if there is no other feedbacks.

Thanks,
Dian

> 在 2019年11月22日，下午2:51，Hequn Cheng  写道：
> 
> Thanks a lot for putting this together, Dian! Definitely +1 for this!
> It is great to make sure that the resources used by the Python process are
> managed properly by Flink’s resource management framework.
> 
> Also, thanks to the guys that working on the unified memory management
> framework.
> 
> Best, Hequn
> 
> 
> On Mon, Nov 18, 2019 at 5:23 PM Yangze Guo  wrote:
> 
>> Thanks for driving this discussion, Dian!
>> 
>> +1 for this proposal. It will help to reduce container failure due to
>> the memory overuse.
>> Some comments left in the design doc.
>> 
>> Best,
>> Yangze Guo
>> 
>> On Mon, Nov 18, 2019 at 4:06 PM Xintong Song 
>> wrote:
>>> 
>>> Sorry for the late reply.
>>> 
>>> +1 for the general proposal.
>>> 
>>> And one remainder, to use UNKNOWN resource requirement, we need to make
>>> sure optimizer knowns which operators use off-heap managed memory, and
>>> compute and set a fraction to the operators. See FLIP-53[1] for more
>>> details, and I would suggest you to double check with @Zhu Zhu who works
>> on
>>> this part.
>>> 
>>> Thank you~
>>> 
>>> Xintong Song
>>> 
>>> 
>>> [1]
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
>>> 
>>> On Tue, Nov 12, 2019 at 11:53 AM Dian Fu  wrote:
>>> 
 Hi Jincheng,
 
 Thanks for the reply and also looking forward to the feedback from the
 community.
 
 Thanks,
 Dian
 
> 在 2019年11月11日，下午2:34，jincheng sun  写道：
> 
> Hi all,
> 
> +1， Thanks for bring up this discussion Dian!
> 
> The Resource Management is very important for PyFlink UDF. So, It's
>> great
> if anyone can add more comments or inputs in the design doc or
>> feedback
 in
> ML. :)
> 
> Best,
> Jincheng
> 
> Dian Fu  于2019年11月5日周二 上午11:32写道：
> 
>> Hi everyone,
>> 
>> In FLIP-58[1] it will add the support of Python user-defined
>> stateless
>> function for Python Table API. It will launch a separate Python
>> process
 for
>> Python user-defined function execution. The resources used by the
>> Python
>> process should be managed properly by Flink’s resource management
>> framework. FLIP-49[2] has proposed a unified memory management
>> framework
>> and PyFlink user-defined function resource management should be
>> based on
>> it. Jincheng, Hequn, Xintong, GuoWei and I discussed offline about
 this. I
>> draft a design doc[3] and want to start a discussion about PyFlink
>> user-defined function resource management.
>> 
>> Welcome any comments on the design doc or giving us feedback on the
>> ML
>> directly.
>> 
>> Regards,
>> Dian
>> 
>> [1]
>> 
 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
>> [2]
>> 
 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
>> [3]
>> 
 
>> https://docs.google.com/document/d/1LQP8L66Thu2yVv6YRSfmF9EkkMnwhBHGjcTQ11GUmFc/edit#heading=h.4q4ggaftf78m
 
 
>> 

Re: [VOTE] Setup a secur...@flink.apache.org mailing list

[Dian Fu]

NOTE: Only PMC votes is binding.

Thanks for sharing your thoughts. I also think that this doesn't fall into any 
of the existing categories listed in the bylaws. Maybe we could do some 
improvements for the bylaws.

This is not codebase change as Robert mentioned and it's related to how to 
manage Flink's development in a good way. So, I agree with Robert and Jincheng 
that this VOTE should only count PMC votes for now. 

Thanks,
Dian

> 在 2019年11月26日，上午11:43，jincheng sun  写道：
> 
> I also think that we should only count PMC votes.
> 
> This ML is to improve the security mechanism for Flink. Of course we don't
> expect to use this
> ML often. I hope that it's perfect if this ML is never used. However, the
> Flink community is growing rapidly, it's better to
> make our security mechanism as convenient as possible. But I agree that
> this ML is not a must to have, it's nice to have.
> 
> So, I give the vote as +1(binding).
> 
> Best,
> Jincheng
> 
> Robert Metzger  于2019年11月25日周一 下午9:45写道：
> 
>> I agree that we are only counting PMC votes (because this decision goes
>> beyond the codebase)
>> 
>> I'm undecided what to vote :) I'm not against setting up a new mailing
>> list, but I also don't think the benefit (having a private list with PMC +
>> committers) is enough to justify the work involved. As far as I remember,
>> we have received 2 security issue notices, both basically about the same
>> issue.  I'll leave it to other PMC members to support this if they want to
>> ...
>> 
>> 
>> On Mon, Nov 25, 2019 at 9:15 AM Dawid Wysakowicz 
>> wrote:
>> 
>>> Hi all,
>>> 
>>> What is the voting scheme for it? I am not sure if it falls into any of
>>> the categories we have listed in our bylaws. Are committers votes
>>> binding or just PMCs'? (Personally I think it should be PMCs') Is this a
>>> binding vote or just an informational vote?
>>> 
>>> Best,
>>> 
>>> Dawid
>>> 
>>> On 25/11/2019 07:34, jincheng sun wrote:
 +1
 
 Dian Fu  于2019年11月21日周四 下午4:11写道：
 
> Hi all,
> 
> According to our previous discussion in [1], I'd like to bring up a
>> vote
> to set up a secur...@flink.apache.org mailing list.
> 
> The vote will be open for at least 72 hours (excluding weekend). I'll
>>> try
> to close it by 2019-11-26 18:00 UTC, unless there is an objection or
>> not
> enough votes.
> 
> Regards,
> Dian
> 
> [1]
> 
>>> 
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Expose-or-setup-a-security-flink-apache-org-mailing-list-for-security-report-and-discussion-tt34950.html#a34951
>>> 
>>> 
>> 

Re: [VOTE] Setup a secur...@flink.apache.org mailing list

[jincheng sun]

 I also think that we should only count PMC votes.

This ML is to improve the security mechanism for Flink. Of course we don't
expect to use this
ML often. I hope that it's perfect if this ML is never used. However, the
Flink community is growing rapidly, it's better to
make our security mechanism as convenient as possible. But I agree that
this ML is not a must to have, it's nice to have.

So, I give the vote as +1(binding).

Best,
Jincheng

Robert Metzger  于2019年11月25日周一 下午9:45写道：

> I agree that we are only counting PMC votes (because this decision goes
> beyond the codebase)
>
> I'm undecided what to vote :) I'm not against setting up a new mailing
> list, but I also don't think the benefit (having a private list with PMC +
> committers) is enough to justify the work involved. As far as I remember,
> we have received 2 security issue notices, both basically about the same
> issue.  I'll leave it to other PMC members to support this if they want to
> ...
>
>
> On Mon, Nov 25, 2019 at 9:15 AM Dawid Wysakowicz 
> wrote:
>
> > Hi all,
> >
> > What is the voting scheme for it? I am not sure if it falls into any of
> > the categories we have listed in our bylaws. Are committers votes
> > binding or just PMCs'? (Personally I think it should be PMCs') Is this a
> > binding vote or just an informational vote?
> >
> > Best,
> >
> > Dawid
> >
> > On 25/11/2019 07:34, jincheng sun wrote:
> > > +1
> > >
> > > Dian Fu  于2019年11月21日周四 下午4:11写道：
> > >
> > >> Hi all,
> > >>
> > >> According to our previous discussion in [1], I'd like to bring up a
> vote
> > >> to set up a secur...@flink.apache.org mailing list.
> > >>
> > >> The vote will be open for at least 72 hours (excluding weekend). I'll
> > try
> > >> to close it by 2019-11-26 18:00 UTC, unless there is an objection or
> not
> > >> enough votes.
> > >>
> > >> Regards,
> > >> Dian
> > >>
> > >> [1]
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Expose-or-setup-a-security-flink-apache-org-mailing-list-for-security-report-and-discussion-tt34950.html#a34951
> >
> >
>

Re: [DISCUSS] Remove old WebUI

[tison]

+1 to drop the old UI.

Best,
tison.


vino yang  于2019年11月25日周一 下午8:28写道：

> +1
>
> Best,
> Vino
>
> Zhu Zhu  于2019年11月25日周一 下午2:06写道：
>
> > +1
> >
> > Thanks,
> > Zhu Zhu
> >
> > Dian Fu  于2019年11月25日周一 上午11:09写道：
> >
> > > +1 to drop the old UI.
> > >
> > > > 在 2019年11月25日，上午10:59，Zhenghua Gao  写道：
> > > >
> > > > +1 to drop the old one.
> > > >
> > > > *Best Regards,*
> > > > *Zhenghua Gao*
> > > >
> > > >
> > > > On Thu, Nov 21, 2019 at 8:05 PM Chesnay Schepler  >
> > > wrote:
> > > >
> > > >> Hello everyone,
> > > >>
> > > >> Flink 1.9 shipped with a new UI, with the old one being kept around
> > as a
> > > >> backup in case something wasn't working as expected.
> > > >>
> > > >> Currently there are no issues indicating any significant problems
> > > >> (exclusive to the new UI), so I wanted to check what people think
> > about
> > > >> dropping the old UI for 1.10.
> > > >>
> > > >>
> > >
> > >
> >
>

Re:[DISCUSS] Multi-topics consuming from KafkaTableSource

[Jun Zhang]

If the data formats of the Multi-topics are different, what do you plan to do?  
 

 
 
 
 On 11/26/2019 08:53??leohttps://issues.apache.org/jira/browse/FLINK-14729


Best,

Leo

[jira] [Created] (FLINK-14944) Unstable test FlinkFnExecutionSyncTests.test_flink_fn_execution_pb2_synced

[Dian Fu (Jira)]

Dian Fu created FLINK-14944:
---

 Summary: Unstable test 
FlinkFnExecutionSyncTests.test_flink_fn_execution_pb2_synced
 Key: FLINK-14944
 URL: https://issues.apache.org/jira/browse/FLINK-14944
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.10.0
Reporter: Dian Fu
 Fix For: 1.10.0


This tests failed occasionally:
{code:java}
force = 'True', output_dir = '/tmp/tmpkh6rmeig'

def generate_proto_files(force=True, output_dir=DEFAULT_PYTHON_OUTPUT_PATH):
try:
import grpc_tools  # noqa  # pylint: disable=unused-import
except ImportError:
warnings.warn('Installing grpcio-tools is recommended for 
development.')

proto_dirs = [os.path.join(PYFLINK_ROOT_PATH, path) for path in 
PROTO_PATHS]
proto_files = sum(
[glob.glob(os.path.join(d, '*.proto')) for d in proto_dirs], [])
out_dir = os.path.join(PYFLINK_ROOT_PATH, output_dir)
out_files = [path for path in glob.glob(os.path.join(out_dir, 
'*_pb2.py'))]

if out_files and not proto_files and not force:
# We have out_files but no protos; assume they're up to date.
# This is actually the common case (e.g. installation from an 
sdist).
logging.info('No proto files; using existing generated files.')
return

elif not out_files and not proto_files:
raise RuntimeError(
'No proto files found in %s.' % proto_dirs)

# Regenerate iff the proto files or this file are newer.
elif force or not out_files or len(out_files) < len(proto_files) or (
min(os.path.getmtime(path) for path in out_files)
<= max(os.path.getmtime(path)
   for path in proto_files + [os.path.realpath(__file__)])):
try:
>   from grpc_tools import protoc
E   ModuleNotFoundError: No module named 'grpc_tools'

pyflink/gen_protos.py:70: ModuleNotFoundError

During handling of the above exception, another exception occurred:

self = 


def test_flink_fn_execution_pb2_synced(self):
>   generate_proto_files('True', self.tempdir)

pyflink/fn_execution/tests/test_flink_fn_execution_pb2_synced.py:35: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyflink/gen_protos.py:83: in generate_proto_files
target=_install_grpcio_tools_and_generate_proto_files(force, output_dir))
pyflink/gen_protos.py:131: in _install_grpcio_tools_and_generate_proto_files
'--upgrade', GRPC_TOOLS, "-I"])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

popenargs = 
(['/home/travis/build/apache/flink/flink-python/.tox/py36/bin/python', '-m', 
'pip', 'install', '--prefix', 
'/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels', 
...],)
kwargs = {}, retcode = 2
cmd = ['/home/travis/build/apache/flink/flink-python/.tox/py36/bin/python', 
'-m', 'pip', 'install', '--prefix', 
'/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels', 
...]

def check_call(*popenargs, **kwargs):
"""Run command with arguments.  Wait for command to complete.  If
the exit code was zero then return, otherwise raise
CalledProcessError.  The CalledProcessError object will have the
return code in the returncode attribute.

The arguments are the same as for the call function.  Example:

check_call(["ls", "-l"])
"""
retcode = call(*popenargs, **kwargs)
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
>   raise CalledProcessError(retcode, cmd)
E   subprocess.CalledProcessError: Command 
'['/home/travis/build/apache/flink/flink-python/.tox/py36/bin/python', '-m', 
'pip', 'install', '--prefix', 
'/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels', 
'--build', 
'/home/travis/build/apache/flink/flink-python/pyflink/../.eggs/grpcio-wheels-build',
 '--upgrade', 'grpcio-tools>=1.3.5,<=1.14.2', '-I']' returned non-zero exit 
status 2.

dev/.conda/envs/3.6/lib/python3.6/subprocess.py:311: CalledProcessError
{code}

instance: [https://api.travis-ci.org/v3/job/616685590/log.txt]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Releasing Flink 1.8.3

[Hequn Cheng]

Hi,

@Ufuk Celebi  Hi, we are very close now. There is one issue(
FLINK-13995 ) left that
I want to double-check with you guys. Once this is done, we can create the
first RC. I already have some minor comments in the PR
.

@Zhu Zhu  Glad to hear that it is not a blocker. Thank
you.

Best, Hequn

On Mon, Nov 25, 2019 at 5:43 PM Ufuk Celebi  wrote:

> @Hequn: flink-shaded:9.0 is available in Maven central now. I think you
> can go ahead and create the first RC. :-)
>
> On Mon, Nov 25, 2019 at 7:47 AM Zhu Zhu  wrote:
>
>> Hi Hequn,
>>
>> Looks we are not able to merge fix of FLINK-14735 to 1.8 very soon.
>> Given that this fix is for batch job only and batch is not very good in
>> 1.8, I think it is a not blocker of release 1.8.3.
>> So just don't be blocked by it and feel free to cut the RC when other
>> blocking issues are resolved.
>>
>> Thanks,
>> Zhu Zhu
>>
>> Hequn Cheng  于2019年11月23日周六 下午9:08写道：
>>
>> > Hi Zhu Zhu,
>> >
>> > Thanks a lot for letting us know!
>> > We can't cut the first RC right now due to the wait of the flink-shade
>> > release, so go ahead.
>> >
>> > Theoretically, we will cut the first RC of 1.8.3 and vote for it once
>> the
>> > release of flink-shade is done,
>> > but I will try my best to have it in 1.8.3. Hope we can get it on board
>> on
>> > time. :)
>> >
>> > Best, Hequn
>> >
>> > On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu  wrote:
>> >
>> >> Hi Jincheng & Hequn
>> >>
>> >> Thanks for driving the releasing of 1.8.3.
>> >>
>> >> I am now working on FLINK-14735. The fix avoids duplicated input
>> >> checking when scheduling ALL-to-ALL
>> >> connected downstream consumers with ALL input constraints. The
>> duplicated
>> >> checking can cause severe
>> >> performance issues for large scale jobs. So I hope the fix could be
>> >> released with 1.8.3.
>> >>
>> >> The fix is already merged into master, and is now in the process of
>> >> backporting to 1.8.
>> >>
>> >> Thanks,
>> >> Zhu Zhu
>> >>
>> >> Ufuk Celebi  于2019年11月15日周五 下午11:54写道：
>> >>
>> >>> Thanks Chesnay.
>> >>>
>> >>> I'm also +1 to release 1.8.3 asap without the changes for the Jackson
>> >>> version bump and leave those for a future release. Realistically, the
>> >>> flink-shaded release will take until mid next week or end of next
>> week.
>> >>> But
>> >>> please correct me if you think that it should not take that long or
>> it's
>> >>> OK
>> >>> to block the 1.8.3 release on the flink-shaded release.
>> >>>
>> >>> – Ufuk
>> >>>
>> >>>
>> >>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler 
>> >>> wrote:
>> >>>
>> >>> > I've kicked off a discussion about the next flink-shaded release,
>> and
>> >>> > have opened PRs for adding the opt-in profile to 1.8/1.9.
>> >>> >
>> >>> > On 15/11/2019 13:54, Hequn Cheng wrote:
>> >>> > > That's great, thank you very much! Ideally, we can kick off the
>> >>> release
>> >>> > > vote for the first RC of 1.8.3 within next week. :)
>> >>> > >
>> >>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <
>> ches...@apache.org
>> >>> >
>> >>> > wrote:
>> >>> > >
>> >>> > >> I'm not aware of any more planned changes to flink-shaded; so we
>> >>> could
>> >>> > >> start the release right away.
>> >>> > >>
>> >>> > >> On 15/11/2019 13:44, Hequn Cheng wrote:
>> >>> > >>> Hi,
>> >>> > >>>
>> >>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in
>> >>> approach
>> >>> > for
>> >>> > >>> 1.8/1.9.
>> >>> > >>> @Ufuk Thank you for the nice summary.
>> >>> > >>>
>> >>> > >>> Looks good so far except that we need to postpone 1.8.3 a bit to
>> >>> first
>> >>> > >> do a
>> >>> > >>> flink-shaded release.
>> >>> > >>> BTW, @chesnay when would we plan to release the flink-shaded
>> with
>> >>> > >> upgraded
>> >>> > >>> Jackson?
>> >>> > >>>
>> >>> > >>> Best, Hequn
>> >>> > >>>
>> >>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <
>> >>> ches...@apache.org>
>> >>> > >> wrote:
>> >>> >  One small modification: the flink-shaded upgrade does not have
>> to
>> >>> be
>> >>> >  part of the profile; since it is only intended for internal use
>> >>> anyway
>> >>> >  (and thus has limited exposure) we can be pretty sure this
>> doesn't
>> >>> > break
>> >>> >  anything.
>> >>> > 
>> >>> >  On 15/11/2019 12:23, Chesnay Schepler wrote:
>> >>> > > Ufuk's summary is correct.
>> >>> > >
>> >>> > > There's a slight caveat in that we'd also have to bump the
>> >>> > > shade-plugin to 3.1.1 since it otherwise fails on jackson,
>> >>> > > but I have no concerns about this change.
>> >>> > >
>> >>> > > On 15/11/2019 12:19, Ufuk Celebi wrote:
>> >>> > >> The opt-in approach seems reasonable to me. +1 to include the
>> >>> > >> profiles in
>> >>> > >> 1.8 and 1.9 without changing the default versions (including
>> the
>> >>> > >> default
>> >>> > >> version of flink-shaded).
>> >>> > >>
>> >>> > >> As far as I can tell, the n

[DISCUSS] Multi-topics consuming from KafkaTableSource

[leo]

Hi, all.


I want to propose an improvement of KafkaTableSource which can consume 
multiple topics at the same time.


Here is proposal,
https://issues.apache.org/jira/browse/FLINK-14729


Best,

Leo

[jira] [Created] (FLINK-14943) make callback and related members protected so that user can override the callback handling

[Steven Zhen Wu (Jira)]

Steven Zhen Wu created FLINK-14943:
--

 Summary: make callback and related members protected so that user 
can override the callback handling
 Key: FLINK-14943
 URL: https://issues.apache.org/jira/browse/FLINK-14943
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Affects Versions: 1.9.1
Reporter: Steven Zhen Wu


FlinkKafkaProducer's `logFailuresOnly` is either log everything or fail 
everything. We want to selectively drop/skip records on certain exceptions, 
like `RecordTooLargeException`. So we want to override the callback when 
extending FlinkKafkaProducer class. Hence this request of making callback, 
asyncException, and acknowledgeMessage protected.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: set job level TTL

[Yun Tang]

Hi Chen

There existed several problems currently:
1. State TTL cannot be applied to Flink SQL
2. No  job-level state TTL
3. Window state should not be limited to TTL if existed a job-level state

If your goal is to achieve #2, we could introduce a configuration via 
flink-conf.yaml so that we could also achieve #1 but leave window state not 
limited to TTL.
I think this deserves an issue to track this request.

Best
Yun Tang

On 11/25/19, 8:04 AM, "Chen Qin"  wrote:

I am a bit confused. Job level TTL is different from state TTL despite
share same name.

What we want to achieve in Job TTL is basically "run x time => save point
=> shutdown job"


On Sun, Nov 24, 2019 at 9:57 AM Yun Tang  wrote:

> Hi Chen
>
> Currently, state TTL setting is per-state scoped, and there is no
> job-level TTL setting so far. Do you want similar settings in flink-conf 
to
> enable Flink SQL could also benefit from state TTL?
>
> Best
> Yun Tang
>
> On 11/25/19, 12:16 AM, "Chen Qin"  wrote:
>
> Hi there,
>
> We have use case of allowing ad hoc user submit a Flink job (mostly
> SQL)
> and user come back and verify results after period of time (eg 
24hours)
> without manual intervention. The idea is to have something like set 
job
> level TTL configuration to 24hours. Jobmanager will honor 24hours of
> instrument save point and proper shutdown of flink job.
>
> Just want to know if we have job level TTL setting or parity features
> JIRA.
>
>
> Best,
> Chen
>
>
>

[jira] [Created] (FLINK-14942) State Processing API: add an option to make deep copy

[Jun Qin (Jira)]

Jun Qin created FLINK-14942:
---

 Summary: State Processing API: add an option to make deep copy
 Key: FLINK-14942
 URL: https://issues.apache.org/jira/browse/FLINK-14942
 Project: Flink
  Issue Type: Improvement
  Components: API / State Processor
Affects Versions: 1.9.1
Reporter: Jun Qin


Current when a new savepoint is created based on a source savepoint, then there 
are references in the new savepoint to the source savepoint. Here is the [State 
Processing API 
doc|https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/libs/state_processor_api.html]
 says: 
bq. Note: When basing a new savepoint on existing state, the state processor 
api makes a shallow copy of the pointers to the existing operators. This means 
that both savepoints share state and one cannot be deleted without corrupting 
the other!

This JIRA is to request an option to have a deep copy (instead of shallow copy) 
such that the new savepoint is self-contained. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Setup a secur...@flink.apache.org mailing list

[Robert Metzger]

I agree that we are only counting PMC votes (because this decision goes
beyond the codebase)

I'm undecided what to vote :) I'm not against setting up a new mailing
list, but I also don't think the benefit (having a private list with PMC +
committers) is enough to justify the work involved. As far as I remember,
we have received 2 security issue notices, both basically about the same
issue.  I'll leave it to other PMC members to support this if they want to
...


On Mon, Nov 25, 2019 at 9:15 AM Dawid Wysakowicz 
wrote:

> Hi all,
>
> What is the voting scheme for it? I am not sure if it falls into any of
> the categories we have listed in our bylaws. Are committers votes
> binding or just PMCs'? (Personally I think it should be PMCs') Is this a
> binding vote or just an informational vote?
>
> Best,
>
> Dawid
>
> On 25/11/2019 07:34, jincheng sun wrote:
> > +1
> >
> > Dian Fu  于2019年11月21日周四 下午4:11写道：
> >
> >> Hi all,
> >>
> >> According to our previous discussion in [1], I'd like to bring up a vote
> >> to set up a secur...@flink.apache.org mailing list.
> >>
> >> The vote will be open for at least 72 hours (excluding weekend). I'll
> try
> >> to close it by 2019-11-26 18:00 UTC, unless there is an objection or not
> >> enough votes.
> >>
> >> Regards,
> >> Dian
> >>
> >> [1]
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Expose-or-setup-a-security-flink-apache-org-mailing-list-for-security-report-and-discussion-tt34950.html#a34951
>
>

Re: [DISCUSS] Remove old WebUI

[vino yang]

+1

Best,
Vino

Zhu Zhu  于2019年11月25日周一 下午2:06写道：

> +1
>
> Thanks,
> Zhu Zhu
>
> Dian Fu  于2019年11月25日周一 上午11:09写道：
>
> > +1 to drop the old UI.
> >
> > > 在 2019年11月25日，上午10:59，Zhenghua Gao  写道：
> > >
> > > +1 to drop the old one.
> > >
> > > *Best Regards,*
> > > *Zhenghua Gao*
> > >
> > >
> > > On Thu, Nov 21, 2019 at 8:05 PM Chesnay Schepler 
> > wrote:
> > >
> > >> Hello everyone,
> > >>
> > >> Flink 1.9 shipped with a new UI, with the old one being kept around
> as a
> > >> backup in case something wasn't working as expected.
> > >>
> > >> Currently there are no issues indicating any significant problems
> > >> (exclusive to the new UI), so I wanted to check what people think
> about
> > >> dropping the old UI for 1.10.
> > >>
> > >>
> >
> >
>

[jira] [Created] (FLINK-14941) The AbstractTableInputFormat#nextRecord in hbase connector will handle the same rowkey twice once encountered any exception

[Zheng Hu (Jira)]

Zheng Hu created FLINK-14941:


 Summary: The AbstractTableInputFormat#nextRecord in hbase 
connector will handle the same rowkey twice once encountered any exception
 Key: FLINK-14941
 URL: https://issues.apache.org/jira/browse/FLINK-14941
 Project: Flink
  Issue Type: Bug
Reporter: Zheng Hu


In the mail list [1].   The user complain that it will see the same row twice 
if encountered any HBase exception. 
The problem is here: 
{code}
public T nextRecord(T reuse) throws IOException {
if (resultScanner == null) {
throw new IOException("No table result scanner 
provided!");
}
try {
Result res = resultScanner.next();
if (res != null) {
scannedRows++;
currentRow = res.getRow();
return mapResultToOutType(res);
}
} catch (Exception e) {
resultScanner.close();
//workaround for timeout on scan
LOG.warn("Error after scan of " + scannedRows + " rows. 
Retry with a new scanner...", e);
scan.setStartRow(currentRow);
resultScanner = table.getScanner(scan);
Result res = resultScanner.next();
if (res != null) {
scannedRows++;
currentRow = res.getRow();
return mapResultToOutType(res);
}
}

endReached = true;
return null;
}
{code}

We will set the startRow of the new scan to the currentRow which has been seen, 
 that means the currentRow will be seen twice.   Actually, we should replace 
the scan.setStartRow(currentRow) as scan.withStartRow(currentRow, false) , the 
false means exclude the currentRow. 


[1]. 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DataSet-API-HBase-ScannerTimeoutException-and-double-Result-processing-td31174.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Releasing Flink 1.8.3

[Ufuk Celebi]

@Hequn: flink-shaded:9.0 is available in Maven central now. I think you can
go ahead and create the first RC. :-)

On Mon, Nov 25, 2019 at 7:47 AM Zhu Zhu  wrote:

> Hi Hequn,
>
> Looks we are not able to merge fix of FLINK-14735 to 1.8 very soon.
> Given that this fix is for batch job only and batch is not very good in
> 1.8, I think it is a not blocker of release 1.8.3.
> So just don't be blocked by it and feel free to cut the RC when other
> blocking issues are resolved.
>
> Thanks,
> Zhu Zhu
>
> Hequn Cheng  于2019年11月23日周六 下午9:08写道：
>
> > Hi Zhu Zhu,
> >
> > Thanks a lot for letting us know!
> > We can't cut the first RC right now due to the wait of the flink-shade
> > release, so go ahead.
> >
> > Theoretically, we will cut the first RC of 1.8.3 and vote for it once the
> > release of flink-shade is done,
> > but I will try my best to have it in 1.8.3. Hope we can get it on board
> on
> > time. :)
> >
> > Best, Hequn
> >
> > On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu  wrote:
> >
> >> Hi Jincheng & Hequn
> >>
> >> Thanks for driving the releasing of 1.8.3.
> >>
> >> I am now working on FLINK-14735. The fix avoids duplicated input
> >> checking when scheduling ALL-to-ALL
> >> connected downstream consumers with ALL input constraints. The
> duplicated
> >> checking can cause severe
> >> performance issues for large scale jobs. So I hope the fix could be
> >> released with 1.8.3.
> >>
> >> The fix is already merged into master, and is now in the process of
> >> backporting to 1.8.
> >>
> >> Thanks,
> >> Zhu Zhu
> >>
> >> Ufuk Celebi  于2019年11月15日周五 下午11:54写道：
> >>
> >>> Thanks Chesnay.
> >>>
> >>> I'm also +1 to release 1.8.3 asap without the changes for the Jackson
> >>> version bump and leave those for a future release. Realistically, the
> >>> flink-shaded release will take until mid next week or end of next week.
> >>> But
> >>> please correct me if you think that it should not take that long or
> it's
> >>> OK
> >>> to block the 1.8.3 release on the flink-shaded release.
> >>>
> >>> – Ufuk
> >>>
> >>>
> >>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler 
> >>> wrote:
> >>>
> >>> > I've kicked off a discussion about the next flink-shaded release, and
> >>> > have opened PRs for adding the opt-in profile to 1.8/1.9.
> >>> >
> >>> > On 15/11/2019 13:54, Hequn Cheng wrote:
> >>> > > That's great, thank you very much! Ideally, we can kick off the
> >>> release
> >>> > > vote for the first RC of 1.8.3 within next week. :)
> >>> > >
> >>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <
> ches...@apache.org
> >>> >
> >>> > wrote:
> >>> > >
> >>> > >> I'm not aware of any more planned changes to flink-shaded; so we
> >>> could
> >>> > >> start the release right away.
> >>> > >>
> >>> > >> On 15/11/2019 13:44, Hequn Cheng wrote:
> >>> > >>> Hi,
> >>> > >>>
> >>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in
> >>> approach
> >>> > for
> >>> > >>> 1.8/1.9.
> >>> > >>> @Ufuk Thank you for the nice summary.
> >>> > >>>
> >>> > >>> Looks good so far except that we need to postpone 1.8.3 a bit to
> >>> first
> >>> > >> do a
> >>> > >>> flink-shaded release.
> >>> > >>> BTW, @chesnay when would we plan to release the flink-shaded with
> >>> > >> upgraded
> >>> > >>> Jackson?
> >>> > >>>
> >>> > >>> Best, Hequn
> >>> > >>>
> >>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <
> >>> ches...@apache.org>
> >>> > >> wrote:
> >>> >  One small modification: the flink-shaded upgrade does not have
> to
> >>> be
> >>> >  part of the profile; since it is only intended for internal use
> >>> anyway
> >>> >  (and thus has limited exposure) we can be pretty sure this
> doesn't
> >>> > break
> >>> >  anything.
> >>> > 
> >>> >  On 15/11/2019 12:23, Chesnay Schepler wrote:
> >>> > > Ufuk's summary is correct.
> >>> > >
> >>> > > There's a slight caveat in that we'd also have to bump the
> >>> > > shade-plugin to 3.1.1 since it otherwise fails on jackson,
> >>> > > but I have no concerns about this change.
> >>> > >
> >>> > > On 15/11/2019 12:19, Ufuk Celebi wrote:
> >>> > >> The opt-in approach seems reasonable to me. +1 to include the
> >>> > >> profiles in
> >>> > >> 1.8 and 1.9 without changing the default versions (including
> the
> >>> > >> default
> >>> > >> version of flink-shaded).
> >>> > >>
> >>> > >> As far as I can tell, the next steps would be:
> >>> > >>
> >>> > >> 1) Release flink-shaded with upgraded Jackson
> >>> > >> 2a) Bump the flink-shaded version by default in master
> >>> > >> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in
> profiles
> >>> > >> should also
> >>> > >> cover the upgrade to the most recent flink-shaded version)
> >>> > >>
> >>> > >> @Chesnay: is this a correct summary?
> >>> > >>
> >>> > >> Note this would block the 1.8.3 release on step 1. As an
> >>> upside, we
> >>> > >> might
> >>> > >> get some additional feedback until the 1.

[jira] [Created] (FLINK-14940) Travis build passes despite Test failures

[Gary Yao (Jira)]

Gary Yao created FLINK-14940:


 Summary: Travis build passes despite Test failures
 Key: FLINK-14940
 URL: https://issues.apache.org/jira/browse/FLINK-14940
 Project: Flink
  Issue Type: Bug
  Components: Test Infrastructure
Affects Versions: 1.10.0
Reporter: Gary Yao


Build https://travis-ci.org/apache/flink/jobs/616462870 is green despite the 
presence of Test failures.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14939) StreamingKafkaITCase fails due to distDir property not being set

[Gary Yao (Jira)]

Gary Yao created FLINK-14939:


 Summary: StreamingKafkaITCase fails due to distDir property not 
being set
 Key: FLINK-14939
 URL: https://issues.apache.org/jira/browse/FLINK-14939
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka, Tests
Affects Versions: 1.10.0
Reporter: Gary Yao


https://api.travis-ci.org/v3/job/616462870/log.txt

{noformat}
08:12:34.965 [INFO] ---
08:12:34.965 [INFO]  T E S T S
08:12:34.965 [INFO] ---
08:12:35.868 [INFO] Running 
org.apache.flink.tests.util.kafka.StreamingKafkaITCase
08:12:35.893 [ERROR] Tests run: 3, Failures: 3, Errors: 0, Skipped: 0, Time 
elapsed: 0.02 s <<< FAILURE! - in 
org.apache.flink.tests.util.kafka.StreamingKafkaITCase
08:12:35.893 [ERROR] testKafka[0: 
kafka-version:0.10.2.0](org.apache.flink.tests.util.kafka.StreamingKafkaITCase) 
 Time elapsed: 0.009 s  <<< FAILURE!
java.lang.AssertionError: The distDir property was not set. You can set it when 
running maven via -DdistDir= .
at 
org.apache.flink.tests.util.kafka.StreamingKafkaITCase.(StreamingKafkaITCase.java:71)

08:12:35.893 [ERROR] testKafka[1: 
kafka-version:0.11.0.2](org.apache.flink.tests.util.kafka.StreamingKafkaITCase) 
 Time elapsed: 0.001 s  <<< FAILURE!
java.lang.AssertionError: The distDir property was not set. You can set it when 
running maven via -DdistDir= .
at 
org.apache.flink.tests.util.kafka.StreamingKafkaITCase.(StreamingKafkaITCase.java:71)

08:12:35.893 [ERROR] testKafka[2: 
kafka-version:2.2.0](org.apache.flink.tests.util.kafka.StreamingKafkaITCase)  
Time elapsed: 0.001 s  <<< FAILURE!
java.lang.AssertionError: The distDir property was not set. You can set it when 
running maven via -DdistDir= .
at 
org.apache.flink.tests.util.kafka.StreamingKafkaITCase.(StreamingKafkaITCase.java:71)

08:12:36.233 [INFO] 
08:12:36.233 [INFO] Results:
08:12:36.233 [INFO] 
08:12:36.233 [ERROR] Failures: 
08:12:36.233 [ERROR]   StreamingKafkaITCase.:71 The distDir property was 
not set. You can set it when running maven via -DdistDir= .
08:12:36.233 [ERROR]   StreamingKafkaITCase.:71 The distDir property was 
not set. You can set it when running maven via -DdistDir= .
08:12:36.233 [ERROR]   StreamingKafkaITCase.:71 The distDir property was 
not set. You can set it when running maven via -DdistDir= .
08:12:36.233 [INFO] 
08:12:36.233 [ERROR] Tests run: 3, Failures: 3, Errors: 0, Skipped: 0
08:12:36.233 [INFO] 
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14938) Flink elasticsearch failure handler re-add indexrequest causes ConcurrentModificationException

[Shengnan YU (Jira)]

Shengnan YU created FLINK-14938:
---

 Summary: Flink elasticsearch failure handler re-add indexrequest 
causes ConcurrentModificationException
 Key: FLINK-14938
 URL: https://issues.apache.org/jira/browse/FLINK-14938
 Project: Flink
  Issue Type: Bug
  Components: Connectors / ElasticSearch
Affects Versions: 1.8.1
Reporter: Shengnan YU


 

When use Elasticsearch connector failure handler (from official example) to 
re-add documents, Flink encountered ConcurrentModificationException.
{code:java}
input.addSink(new ElasticsearchSink<>(
config, transportAddresses,
new ElasticsearchSinkFunction() {...},
new ActionRequestFailureHandler() {
@Override
void onFailure(ActionRequest action,
Throwable failure,
int restStatusCode,
RequestIndexer indexer) throw Throwable {

if (ExceptionUtils.findThrowable(failure, 
EsRejectedExecutionException.class).isPresent()) {
// full queue; re-add document for indexing
indexer.add(action);
}
}
}));
{code}
I found that in method BufferingNoOpRequestIndexer$processBufferedRequests, it 
will iterator a list of ActionRequest. However the failure handler will keep 
re-adding request to that list after bulk, which causes 
ConcurrentModificationException.

I think it should be a multi-thread bug and need to find a thread-safe List to 
maintain the failure request?

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14937) Flink WebUI could not display

[Yang Wang (Jira)]

Yang Wang created FLINK-14937:
-

 Summary: Flink WebUI could not display
 Key: FLINK-14937
 URL: https://issues.apache.org/jira/browse/FLINK-14937
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Web Frontend
Reporter: Yang Wang
 Attachments: image-2019-11-25-16-21-51-727.png, 
image-2019-11-25-16-23-10-618.png

Both standalone and Yarn Flink webui could not display.

!image-2019-11-25-16-21-51-727.png!

However, the api server seems display normally.

!image-2019-11-25-16-23-10-618.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Setup a secur...@flink.apache.org mailing list

[Dawid Wysakowicz]

Hi all,

What is the voting scheme for it? I am not sure if it falls into any of
the categories we have listed in our bylaws. Are committers votes
binding or just PMCs'? (Personally I think it should be PMCs') Is this a
binding vote or just an informational vote?

Best,

Dawid

On 25/11/2019 07:34, jincheng sun wrote:
> +1
>
> Dian Fu  于2019年11月21日周四 下午4:11写道：
>
>> Hi all,
>>
>> According to our previous discussion in [1], I'd like to bring up a vote
>> to set up a secur...@flink.apache.org mailing list.
>>
>> The vote will be open for at least 72 hours (excluding weekend). I'll try
>> to close it by 2019-11-26 18:00 UTC, unless there is an objection or not
>> enough votes.
>>
>> Regards,
>> Dian
>>
>> [1]
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Expose-or-setup-a-security-flink-apache-org-mailing-list-for-security-report-and-discussion-tt34950.html#a34951



signature.asc
Description: OpenPGP digital signature