Good work for flink's batch processing!
Remote shuffle service can resolve the container lost problem and reduce
the running time for batch jobs once failover. We have investigated the
component a lot and welcome Flink's native solution. We will try it and
help improve it.
Thanks,
Liu Jiangang
Can you please elaborare on what has solved your issue, so others that run
into it could learn from your experience?
Best,
D.
On Wed 1. 12. 2021 at 3:38, chenqizhu wrote:
> Hi,
>
> My problem has been solved. Thank you again
>
> Best regards
>
> 在 2021-12-01 09:58:52，"chenqizhu" 写道：
>
> Hi
Very thanks for all the warm responses ! We are greatly welcome more use cases
and co-work on Flink Remote Shuffle and bash processing with Flink~
Best,
Yun
--
From:Yingjie Cao
Send Time:2021 Dec. 1 (Wed.) 11:16
To:dev
shizhengchao created FLINK-25124:
Summary: A deadlock occurs when the jdbc sink uses two consecutive
dimension tables to associate
Key: FLINK-25124
URL: https://issues.apache.org/jira/browse/FLINK-25124
Wenlong Lyu created FLINK-25123:
---
Summary: Improve expression description in SQL operator
Key: FLINK-25123
URL: https://issues.apache.org/jira/browse/FLINK-25123
Project: Flink
Issue Type:
Thanks for the explanation Matthias. The solution sounds good to me.
I have no more concerns and +1 for the FLIP.
Thanks,
Zhu
Xintong Song 于2021年12月1日周三 下午12:56写道：
> @David,
>
> Thanks for the clarification.
>
> No more concerns from my side. +1 for this FLIP.
>
> Thank you~
>
> Xintong Song
>
L Z created FLINK-25122:
---
Summary: flink-dist/src/main/flink-bin/bin/flink does not expand
variable for FLINK_ENV_JAVA_OPTS
Key: FLINK-25122
URL: https://issues.apache.org/jira/browse/FLINK-25122
Project:
@David,
Thanks for the clarification.
No more concerns from my side. +1 for this FLIP.
Thank you~
Xintong Song
On Wed, Dec 1, 2021 at 12:28 AM Till Rohrmann wrote:
> Given the other breaking changes, I think that it is ok to remove the
> `RunningJobsRegistry` completely.
>
> Since we
Huang Xingbo created FLINK-25121:
Summary: Support ML Python API
Key: FLINK-25121
URL: https://issues.apache.org/jira/browse/FLINK-25121
Project: Flink
Issue Type: New Feature
Huang Xingbo created FLINK-25120:
Summary: Add many kinds of checks in ML Python API
Key: FLINK-25120
URL: https://issues.apache.org/jira/browse/FLINK-25120
Project: Flink
Issue Type: New
Cassie created FLINK-25119:
--
Summary: The name of freeSlots parameter in
SlotSharingSlotAllocator#determineParallelism method is named incorrectly.
Key: FLINK-25119
URL: https://issues.apache.org/jira/browse/FLINK-25119
Hi Jing,
Great to hear that, collaborations and feedbacks are welcomed.
Best,
Yingjie
Jing Zhang 于2021年12月1日周三 上午10:34写道：
> Amazing!
> Remote shuffle service is an important improvement for batch data
> processing experience on Flink.
> It is also a strong requirement in our internal batch
Hi Martijn,
Yes, we will add it to flink-packages soon.
Best,
Yingjie
Martijn Visser 于2021年12月1日周三 上午3:24写道：
> Hi Yingjie,
>
> This is great, thanks for sharing. Will you also add it to
> https://flink-packages.org/ ?
>
> Best regards,
>
> Martijn
>
> On Tue, 30 Nov 2021 at 17:31, Till
Wenlong Lyu created FLINK-25118:
---
Summary: Add vertex index as prefix in vertex name
Key: FLINK-25118
URL: https://issues.apache.org/jira/browse/FLINK-25118
Project: Flink
Issue Type: Sub-task
zzt created FLINK-25117:
---
Summary: NoSuchMethodError getCatalog()
Key: FLINK-25117
URL: https://issues.apache.org/jira/browse/FLINK-25117
Project: Flink
Issue Type: Bug
Components: Table SQL
Amazing!
Thanks Yingjie and all contributors for your great work.
Best,
Jingsong
On Wed, Dec 1, 2021 at 10:52 AM Yun Tang wrote:
>
> Great news!
> Thanks for all the guys who contributed in this project.
>
> Best
> Yun Tang
>
> On 2021/11/30 16:30:52 Till Rohrmann wrote:
> > Great news,
Yun Tang created FLINK-25116:
Summary: Fabric8FlinkKubeClientITCase hangs on Azure
Key: FLINK-25116
URL: https://issues.apache.org/jira/browse/FLINK-25116
Project: Flink
Issue Type: Bug
Amazing!
Remote shuffle service is an important improvement for batch data
processing experience on Flink.
It is also a strong requirement in our internal batch business, we would
try it soon and give you feedback.
Best,
Jing Zhang
Martijn Visser 于2021年12月1日周三 上午3:25写道：
> Hi Yingjie,
>
> This
hi, @Chesnay Schepler would you explain more about
what would happen when deprecating Java 8, but still support it. IMO, if we
still generate packages based on Java 8 which seems to be a consensus, we
still can not take the advantages you mentioned even if we announce that
Java 8 support is
Hi Yingjie,
This is great, thanks for sharing. Will you also add it to
https://flink-packages.org/ ?
Best regards,
Martijn
On Tue, 30 Nov 2021 at 17:31, Till Rohrmann wrote:
> Great news, Yingjie. Thanks a lot for sharing this information with the
> community and kudos to all the
Thanks everyone for your votes!
The proposal has passed with the following votes:
+1 (Binding): 3 (Jiangjie, Yun, Dian)
+1 (Non-binding): 0
-1: 0
Cheers,
Dong
On Tue, Nov 30, 2021 at 6:21 PM Dian Fu wrote:
> +1
>
> Regards,
> Dian
>
> On Tue, Nov 30, 2021 at 12:08 PM Yun Gao
> wrote:
>
> >
Great news, Yingjie. Thanks a lot for sharing this information with the
community and kudos to all the contributors of the external shuffle service
:-)
Cheers,
Till
On Tue, Nov 30, 2021 at 2:32 PM Yingjie Cao wrote:
> Hi dev & users,
>
> We are happy to announce the open source of remote
Given the other breaking changes, I think that it is ok to remove the
`RunningJobsRegistry` completely.
Since we allow users to specify a HighAvailabilityServices implementation
when starting Flink via `high-availability: FQDN`, I think we should mark
the interface at least @Experimental.
hjw created FLINK-25115:
---
Summary: Why Flink Sink operator metric numRecordsOut and
numRecordsOutPerSecond always equal 0
Key: FLINK-25115
URL: https://issues.apache.org/jira/browse/FLINK-25115
Project: Flink
Francesco Guardiani created FLINK-25114:
---
Summary: Remove flink-scala dependency from flink-table-runtime
Key: FLINK-25114
URL: https://issues.apache.org/jira/browse/FLINK-25114
Project: Flink
Hi dev & users,
We are happy to announce the open source of remote shuffle project [1] for
Flink. The project is originated in Alibaba and the main motivation is to
improve batch data processing for both performance & stability and further
embrace cloud native. For more features about the
Response to Wenlongs's feedback:
> I would prefer not to provide such a shortcut, let users use COMPILE
PLAN IF NOT EXISTS and EXECUTE explicitly, which can be understood by
new users even without inferring the docs.
I would like to hear more opinions on this topic. Personally, I find a
Hi Till,
We thought that breaking interfaces, specifically
HighAvailabilityServices and RunningJobsRegistry, was acceptable in this
instance because:
- Neither of these interfaces are marked @Public and so carry no
guarantees about being public and stable.
- As far as we are aware, we
Hi everyone,
thanks for the feedback so far. Let me answer each email indvidually.
I will start with a response to Ingo's feedback:
> Will the JSON plan's schema be considered an API?
No, not in the first version. This is explicitly mentioned in the
`General JSON Plan Assumptions`. I tried
Francesco Guardiani created FLINK-25113:
---
Summary: Cleanup from Parquet and Orc the partition key handling
logic
Key: FLINK-25113
URL: https://issues.apache.org/jira/browse/FLINK-25113
Project:
Chesnay Schepler created FLINK-25112:
Summary: Remove TTL from e2e cache
Key: FLINK-25112
URL: https://issues.apache.org/jira/browse/FLINK-25112
Project: Flink
Issue Type: Technical Debt
Hi Xintong,
However, it's probably not so good for users who don't need such
> retrieval and already used a ZooKeeper/Native-Kubernetes HA to specify
> another remote FS path for storing job results, even if they are
> automatically cleaned-up on committed.
>
Users of ZK / k8s HA are forced to
Marios Trivyzas created FLINK-25111:
---
Summary: Introduce config option, to keep previous behaviour
Key: FLINK-25111
URL: https://issues.apache.org/jira/browse/FLINK-25111
Project: Flink
+1 (binding)
Best,
Godfrey
Jark Wu 于2021年11月30日周二 下午5:47写道：
>
> Thanks for the great discussion and updating.
> Still +1 from my side.
>
> Best,
> Jark
>
> On Tue, 30 Nov 2021 at 17:27, Kurt Young wrote:
>
> > +1 from my side.
> >
> > Best,
> > Kurt
> >
> >
> > On Tue, Nov 30, 2021 at 5:12 PM
sky created FLINK-25110:
---
Summary: Flink SQL consumes Kafka up to a certain point in time
Key: FLINK-25110
URL: https://issues.apache.org/jira/browse/FLINK-25110
Project: Flink
Issue Type: Improvement
Sergey Nuyanzin created FLINK-25109:
---
Summary: Update jline3 to 3.21.0
Key: FLINK-25109
URL: https://issues.apache.org/jira/browse/FLINK-25109
Project: Flink
Issue Type: Sub-task
+1
Regards,
Dian
On Tue, Nov 30, 2021 at 12:08 PM Yun Gao
wrote:
> +1 for the new GraphBuilder API supports more complex
> estimator / model structure and helps to reduce the repeat
> code. Thanks Dong for the proposal.
>
> Best,
> Yun
>
>
>
Thanks for the explanations, Matthias.
Including JobResultStore in HighAvailabilityServices as a replacement of
RunningJobRegistry makes sense to me. And initializing JobResultStore in
the same way initializing JobGraphStore also sounds good.
I have another question concerning where to persist
Hi Kurt,
thanks for sharing your concerns. Our naming is based on the fact that
there is already a JobResult class. That's the metadata container we store
in the JobResultStore. That JobResult is furthermore used in the REST API
where (ironically) it is handled by the JobExecutionResultHandler.
Thanks for the great discussion and updating.
Still +1 from my side.
Best,
Jark
On Tue, 30 Nov 2021 at 17:27, Kurt Young wrote:
> +1 from my side.
>
> Best,
> Kurt
>
>
> On Tue, Nov 30, 2021 at 5:12 PM Jingsong Li
> wrote:
>
> > Hi everyone,
> >
> > Many thanks to Stephan and Timo, this makes
Hi Xintong,
your observation is correct. We probably didn't address this in the FLIP
explicitly enough. We planned to include it in the HighAvailabilityServices
analogously to the RunningJobRegistry (and replace the RunningJobRegistry
by the JobResultStore in the end).
One additional thing, I
+1 from my side.
Best,
Kurt
On Tue, Nov 30, 2021 at 5:12 PM Jingsong Li wrote:
> Hi everyone,
>
> Many thanks to Stephan and Timo, this makes the design of FLIP much
> clearer and more reliable.
>
> I request that you can take another look at the updated FLIP and
> please respond directly if
Hi everyone,
Many thanks to Stephan and Timo, this makes the design of FLIP much
clearer and more reliable.
I request that you can take another look at the updated FLIP and
please respond directly if you have feedback.
(I will contact binding voters directly to confirm)
Best,
Jingsong
On Tue,
Hi Zhu Zhu,
thanks for your reply. Your concern is valid. Our goal is to only touch the
CompletedCheckpointStore and CheckpointIDCounter without instantiating
JobMaster/Scheduler/ExecutionGraph. We would have to initialize these
classes (and for the CompletedCheckpointStore reload the
aresyhzhang created FLINK-25108:
---
Summary: When the environment variable HADOOP_CONF_DIR flink
kerberos authentication error is set
Key: FLINK-25108
URL: https://issues.apache.org/jira/browse/FLINK-25108
I have a question regarding the `COMPILE PLAN OVEWRITE`. If we choose to go
with the config option instead,
that doesn't provide the flexibility to overwrite certain plans but not
others, since the config applies globally, isn't that
something to consider?
On Mon, Nov 29, 2021 at 10:15 AM Marios
Thanks for the healthy discussion. Also +1 from my side for this FLIP.
Thanks,
Timo
On 24.11.21 19:05, Stephan Ewen wrote:
Thanks for all the details and explanation.
With the conclusion of the discussion, also +1 from my side for this FLIP
On Sat, Nov 13, 2021 at 12:23 PM Jingsong Li
Matthias created FLINK-25107:
Summary: GlueSchemaRegistryAvroKinesisITCase and
GlueSchemaRegistryJsonKinesisITCase are skipped on AzureCI but fail when enabled
Key: FLINK-25107
URL:
Varun Yeligar created FLINK-25106:
-
Summary: Support tombstone messages in FLINK's "kafka" connector
Key: FLINK-25106
URL: https://issues.apache.org/jira/browse/FLINK-25106
Project: Flink
49 matches
Mail list logo
