date:20230505

[jira] [Created] (FLINK-32022) Source level scaling is not applied to operators with chaining applied

2023-05-05 Thread Tan Kim (Jira)

Tan Kim created FLINK-32022:
---

 Summary: Source level scaling is not applied to operators with 
chaining applied
 Key: FLINK-32022
 URL: https://issues.apache.org/jira/browse/FLINK-32022
 Project: Flink
  Issue Type: Bug
  Components: Autoscaler
Reporter: Tan Kim


The scaling algorithm is applied on a per-vertex basis, and if the source and 
downstream operators are chained together, source scaling is not applied.
Is this intended?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32021) Improvement the Javadoc for SpecifiedOffsetsInitializer and TimestampOffsetsInitializer.

2023-05-05 Thread Hongshun Wang (Jira)

Hongshun Wang created FLINK-32021:
-

 Summary: Improvement the Javadoc for SpecifiedOffsetsInitializer 
and TimestampOffsetsInitializer.
 Key: FLINK-32021
 URL: https://issues.apache.org/jira/browse/FLINK-32021
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Kafka
Affects Versions: kafka-3.0.0
Reporter: Hongshun Wang
 Fix For: kafka-4.0.0


As described in 
[FLIP-288|https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source],
 Current JavaDoc does not fully explain the behavior of OffsetsInitializers. 
When the partition does not meet the condition, there will be a different 
offset strategy. This may lead to misunderstandings in the design and usage.

 

Add to SpecifiedOffsetsInitializer: "Use Specified offset for specified 
partitions while use commit offset or Earliest for unspecified partitions. 
Specified partition offset should be less than the latest offset, otherwise it 
will start from the earliest."

 

Add to TimestampOffsetsInitializer:Initialize the offsets based on a timestamp. 
If the message meeting the requirement of the timestamp have not been produced 
to Kafka yet, just use the latest offset.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32020) Enable Dynamic Partition Discovery by Default in Kafka Source based on FLIP-288

2023-05-05 Thread Hongshun Wang (Jira)

Hongshun Wang created FLINK-32020:
-

 Summary: Enable Dynamic Partition Discovery by Default in Kafka 
Source based on FLIP-288
 Key: FLINK-32020
 URL: https://issues.apache.org/jira/browse/FLINK-32020
 Project: Flink
  Issue Type: Sub-task
Reporter: Hongshun Wang


As described in 
[FLIP-288|https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source],
 dynamic partition discovery is disabled by default, and users have to specify 
the interval of discovery in order to turn it on.

This subtask is to enable Dynamic Partition Discovery by Default in Kafka 
Source.

Partition discovery is performed on the KafkaSourceEnumerator, which 
asynchronously fetches topic metadata from the Kafka cluster and checks if 
there are any new topics and partitions. This should not cause performance 
issues on the Flink side.

On the Kafka broker side, partition discovery sends a MetadataRequest to the 
Kafka broker to fetch topic information. Considering that the Kafka broker has 
its metadata cache and the default request frequency is relatively low (once 
every 30 seconds), this is not a heavy operation, and the broker's performance 
will not be significantly affected.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32019) EARLIEST offset strategy for partitions discoveried later based on FLIP-288

2023-05-05 Thread Hongshun Wang (Jira)

Hongshun Wang created FLINK-32019:
-

 Summary: EARLIEST offset strategy for partitions discoveried later 
based on FLIP-288
 Key: FLINK-32019
 URL: https://issues.apache.org/jira/browse/FLINK-32019
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Kafka
Affects Versions: kafka-3.0.0
Reporter: Hongshun Wang
 Fix For: kafka-4.0.0


As described in 
[FLIP-288|https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source],
 the strategy used for new partitions is the same as the initial offset 
strategy, which is not reasonable.

According to the semantics, if the startup strategy is latest, the consumed 
data should include all data from the moment of startup, which also includes 
all messages from new created partitions. However, the latest strategy 
currently maybe used for new partitions, leading to the loss of some data 
(thinking a new partition is created and might be discovered by Kafka source 
several minutes later, and the message produced into the partition within the 
gap might be dropped if we use for example "latest" as the initial offset 
strategy).if the data from all new partitions is not read, it does not meet the 
user's expectations.

Other ploblems see final Section of 
[FLIP-288|https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source]:
 {{User specifies OffsetsInitializer for new partition}} .

Therefore, it’s better to provide an *EARLIEST* strategy for later discovered 
partitions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32018) Many builds of benchmark have been interrupted since 20230428

2023-05-05 Thread Yanfei Lei (Jira)

Yanfei Lei created FLINK-32018:
--

 Summary: Many builds of benchmark have been interrupted since 
20230428
 Key: FLINK-32018
 URL: https://issues.apache.org/jira/browse/FLINK-32018
 Project: Flink
  Issue Type: Bug
  Components: Benchmarks
Affects Versions: 1.18.0
Reporter: Yanfei Lei






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32017) DISTINCT COUNT result is incorrect with ttl

2023-05-05 Thread Bo Cui (Jira)

Bo Cui created FLINK-32017:
--

 Summary: DISTINCT COUNT result is incorrect with ttl
 Key: FLINK-32017
 URL: https://issues.apache.org/jira/browse/FLINK-32017
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.17.0, 1.15.0, 1.12.0, 1.18.0
Reporter: Bo Cui


SQL: SELECT COUNT(DISTINCT `c`) FROM Table1
and set ttl to 10s

and Flink will generate code:
{code:java}
public final class GroupAggsHandler$15 implements 
org.apache.flink.table.runtime.generated.AggsHandleFunction {

  long agg0_count;
  boolean agg0_countIsNull;
  private transient 
org.apache.flink.table.runtime.typeutils.ExternalSerializer 
externalSerializer$0;
  private transient 
org.apache.flink.table.runtime.typeutils.ExternalSerializer 
externalSerializer$1;
  private org.apache.flink.table.runtime.dataview.StateMapView 
distinctAcc_0_dataview;
  private org.apache.flink.table.data.binary.BinaryRawValueData 
distinctAcc_0_dataview_raw_value;
  private org.apache.flink.table.api.dataview.MapView distinct_view_0;
  org.apache.flink.table.data.GenericRowData acc$3 = new 
org.apache.flink.table.data.GenericRowData(2);
  org.apache.flink.table.data.GenericRowData acc$5 = new 
org.apache.flink.table.data.GenericRowData(2);
  org.apache.flink.table.data.GenericRowData aggValue$14 = new 
org.apache.flink.table.data.GenericRowData(1);

  private org.apache.flink.table.runtime.dataview.StateDataViewStore 
store;

  public GroupAggsHandler$15(java.lang.Object[] references) throws 
Exception {
externalSerializer$0 = 
(((org.apache.flink.table.runtime.typeutils.ExternalSerializer) references[0]));
externalSerializer$1 = 
(((org.apache.flink.table.runtime.typeutils.ExternalSerializer) references[1]));
  }

  private org.apache.flink.api.common.functions.RuntimeContext 
getRuntimeContext() {
return store.getRuntimeContext();
  }

  @Override
  public void 
open(org.apache.flink.table.runtime.dataview.StateDataViewStore store) throws 
Exception {
this.store = store;

distinctAcc_0_dataview = 
(org.apache.flink.table.runtime.dataview.StateMapView) 
store.getStateMapView("distinctAcc_0", true, externalSerializer$0, 
externalSerializer$1);
distinctAcc_0_dataview_raw_value = 
org.apache.flink.table.data.binary.BinaryRawValueData.fromObject(distinctAcc_0_dataview);
distinct_view_0 = distinctAcc_0_dataview;
  }

  @Override
  public void accumulate(org.apache.flink.table.data.RowData accInput) 
throws Exception {

int field$7;
boolean isNull$7;
boolean isNull$9;
long result$10;
isNull$7 = accInput.isNullAt(0);
field$7 = -1;
if (!isNull$7) {
  field$7 = accInput.getInt(0);
}
java.lang.Integer distinctKey$8 = (java.lang.Integer) field$7;
if (isNull$7) {
  distinctKey$8 = null;
}
 
java.lang.Long value$12 = (java.lang.Long) 
distinct_view_0.get(distinctKey$8);
if (value$12 == null) {
  value$12 = 0L;
}
   
boolean is_distinct_value_changed_0 = false;

long existed$13 = ((long) value$12) & (1L << 0);
if (existed$13 == 0) {  // not existed
  value$12 = ((long) value$12) | (1L << 0);
  is_distinct_value_changed_0 = true;
  
long result$11 = -1L;
boolean isNull$11;
if (isNull$7) {
  
 // --- Cast section generated by 
org.apache.flink.table.planner.functions.casting.IdentityCastRule
 
 // --- End cast section
   
  isNull$11 = agg0_countIsNull;
  if (!isNull$11) {
result$11 = agg0_count;
  }
}
else {
  


isNull$9 = agg0_countIsNull || false;
result$10 = -1L;
if (!isNull$9) {
  

result$10 = (long) (agg0_count + ((long) 1L));

  
}

 // --- Cast section generated by 
org.apache.flink.table.planner.functions.casting.IdentityCastRule
 
 // --- End cast section
   
  isNull$11 = isNull$9;
  if (!isNull$11) {
result$11 = result$10;
  }
}
agg0_count = result$11;;
agg0_countIsNull = isNull$11;
   
}
   
if (is_distinct_value_changed_0) {

[jira] [Created] (FLINK-32016) Logs are printed as disabled even if the autoscaler option is true

2023-05-05 Thread Tan Kim (Jira)

Tan Kim created FLINK-32016:
---

 Summary: Logs are printed as disabled even if the autoscaler 
option is true
 Key: FLINK-32016
 URL: https://issues.apache.org/jira/browse/FLINK-32016
 Project: Flink
  Issue Type: Improvement
  Components: Autoscaler
Reporter: Tan Kim


The part where the job.autoscaler.enabled option is logged as disabled due to 
the code below, even if it is set to true, is misleading to users and needs to 
be corrected in the log message.
{code:java}
if (resource.getSpec().getJob() == null || 
!conf.getBoolean(AUTOSCALER_ENABLED)) {
LOG.info("Job autoscaler is disabled");
return false;
} {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32015) Flink does not fail SSL Handshake when expired cert is used

2023-05-05 Thread Dhruv Patel (Jira)

Dhruv Patel created FLINK-32015:
---

 Summary: Flink does not fail SSL Handshake when expired cert is 
used
 Key: FLINK-32015
 URL: https://issues.apache.org/jira/browse/FLINK-32015
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Reporter: Dhruv Patel


We recently observed that Flink is able to perform SSL communication without 
any issues even if certs in the trust store are expired

As per this issue in Netty library, it says that Default TrustManagerFactory 
does not come with that check in place and we need to implement our own 
TrustManagerFactory for the desired behavior
https://github.com/netty/netty/issues/8461



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32014) Cassandra source documentation is missing

2023-05-05 Thread Etienne Chauchot (Jira)

Etienne Chauchot created FLINK-32014:


 Summary: Cassandra source documentation is missing
 Key: FLINK-32014
 URL: https://issues.apache.org/jira/browse/FLINK-32014
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / Cassandra, Documentation
Reporter: Etienne Chauchot






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32013) Moves DefaultLeaderElectionService.startLeaderElectionBackend() into HAServices

2023-05-05 Thread Matthias Pohl (Jira)

Matthias Pohl created FLINK-32013:
-

 Summary: Moves 
DefaultLeaderElectionService.startLeaderElectionBackend() into HAServices
 Key: FLINK-32013
 URL: https://issues.apache.org/jira/browse/FLINK-32013
 Project: Flink
  Issue Type: Sub-task
Reporter: Matthias Pohl






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32012) Operator failed to rollback due to missing HA metadata

2023-05-05 Thread Nicolas Fraison (Jira)

Nicolas Fraison created FLINK-32012:
---

 Summary: Operator failed to rollback due to missing HA metadata
 Key: FLINK-32012
 URL: https://issues.apache.org/jira/browse/FLINK-32012
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.4.0
Reporter: Nicolas Fraison


The operator has well detected that the job was failing and initiate the 
rollback but this rollback has failed due to `Rollback is not possible due to 
missing HA metadata`

We are relying on saevpoint upgrade mode and zookeeper HA.

The operator is performing a set of action to also delete this HA data in 
savepoint upgrade mode:
 * [flink-kubernetes-operator/AbstractFlinkService.java at main · 
apache/flink-kubernetes-operator|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L346]
 : Suspend job with savepoint and deleteClusterDeployment

 * [flink-kubernetes-operator/StandaloneFlinkService.java at main · 
apache/flink-kubernetes-operator|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/StandaloneFlinkService.java#L158]
 : Remove JM + TM deployment and delete HA data

 * [flink-kubernetes-operator/AbstractFlinkService.java at main · 
apache/flink-kubernetes-operator|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L1008]
 : Wait cluster shutdown and delete zookeeper HA data

 * [flink-kubernetes-operator/FlinkUtils.java at main · 
apache/flink-kubernetes-operator|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/utils/FlinkUtils.java#L155]
 : Remove all child znode

Then when running rollback the operator is looking for HA data even if we rely 
on sevepoint upgrade mode:
 * [flink-kubernetes-operator/AbstractFlinkResourceReconciler.java at main · 
apache/flink-kubernetes-operator|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractFlinkResourceReconciler.java#L164]
 Perform reconcile of rollback if it should rollback

 * [flink-kubernetes-operator/AbstractFlinkResourceReconciler.java at main · 
apache/flink-kubernetes-operator|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractFlinkResourceReconciler.java#L387]
 Rollback failed as HA data is not available

 * [flink-kubernetes-operator/FlinkUtils.java at main · 
apache/flink-kubernetes-operator|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/utils/FlinkUtils.java#L220]
 Check if some child znodes are available

For both step the pattern looks to be the same for kubernetes HA so it doesn't 
looks to be linked to a bug with zookeeper.

 

>From https://issues.apache.org/jira/browse/FLINK-30305 it looks to be expected 
>that the HA data has been deleted (as it is also performed by flink when 
>relying on savepoint upgrade mode)

So I'm wondering why we enforce such a check when performing rollback if we 
rely on savepoint upgrade mode.

Would it be fine to not rely on the HA data and rollback from the last 
savepoint (the one we used in the deployment step)?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32011) flink1.15.2 loaded all the data in the table in mysql5.7

2023-05-05 Thread xueyongyang (Jira)

xueyongyang created FLINK-32011:
---

 Summary: flink1.15.2 loaded all the data in the table in mysql5.7
 Key: FLINK-32011
 URL: https://issues.apache.org/jira/browse/FLINK-32011
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / JDBC
Affects Versions: 1.15.2
 Environment: flink1.15.2

mysql5.7
Reporter: xueyongyang
 Fix For: 1.15.2


CREATE TABLE `T_P_FILTER_MERCHANT_DAY_RES2` (
  `MERCHANT_NO` varchar(200) NOT NULL,
  `ACT_ID` varchar(50) NOT NULL,
  `RULE_ID` varchar(50) NOT NULL,
  `SUM_MONEY` decimal(25,5) DEFAULT NULL,
  `SUM_NUM` decimal(25,5) DEFAULT NULL,
  `DATE_DT` int NOT NULL,
  `DATE_TYPE` varchar(50) DEFAULT NULL,
  `BEGIN_DATE` int DEFAULT NULL,
  `END_DATE` int DEFAULT NULL,
  `ID` bigint NOT NULL AUTO_INCREMENT,
  PRIMARY KEY (`ID`),
  UNIQUE KEY `T_P_FILTER_MERCHANT_DAY_RES2_UN` 
(`MERCHANT_NO`,`ACT_ID`,`RULE_ID`,`DATE_DT`)
) ENGINE=InnoDB AUTO_INCREMENT=88 DEFAULT CHARSET=utf8mb4 
COLLATE=utf8mb4_0900_ai_ci;

 

CREATE TABLE `T_P_RED_STAND_RESULT` (
  `ACTIVITY_NO` varchar(50)  ,
  `RULE_ID` varchar(50)    ,
  `WEIGHT` int NOT NULL ,
  `DEAL_TYPE` varchar(50)  ,
  `DATA_TYPE` varchar(50)  ,
  `DATA_NO` varchar(50) ,
  `USER_TYPE` varchar(50)  ,
  `USER_NO` varchar(50)  ,
  `SUM_MONEY` decimal(25,5) ,
  `SUM_NUM` decimal(25,5) ,
  `DATE_TYPE` varchar(50)  ,
  `BEGIN_TIME` int NOT NULL ,
  `END_TIME` int NOT NULL ,
  `TRADE_MONEY` decimal(25,5) DEFAULT NULL,
  `EXTEND_DATA_MEAN` varchar(50)  ,
  `DATA1` varchar(50)  ,
  `DATA2` varchar(50)  ,
  `DATA3` varchar(50)  ,
  `DATA4` varchar(50)  ,
  `DATA5` varchar(50)  ,
  `CK_REACH` varchar(10)  ,
  `READ_STATUS` varchar(10)  ,
  `PROCESS_STATUS` varchar(50)  ,
  `PROCESS_STATUS_DESC` varchar(200)  ,
  `RESULT_TYPE` varchar(50) ,
  `MANAGER_NO` varchar(50) ,
  `MANAGER_ORG_NO` varchar(50)  ,
  `CALCULATION_DATE` int NOT NULL ,
  `CALCULATION_TIME` datetime DEFAULT NULL ,
  `CREATE_TIME` datetime DEFAULT NULL ,
  `CREATE_USER_NO` varchar(50)  ,
  `MAINTENANCE_TIME` varchar(50)  ,
  `MAINTENANCE_USER_NO` varchar(50)  ,
  `ID` bigint NOT NULL AUTO_INCREMENT,
  PRIMARY KEY (`ID`),
  UNIQUE KEY `T_P_RED_STAND_RESULT_UN` 
(`ACTIVITY_NO`,`USER_NO`,`BEGIN_TIME`,`END_TIME`)
) ENGINE=InnoDB AUTO_INCREMENT=13 DEFAULT CHARSET=utf8mb4 
COLLATE=utf8mb4_0900_ai_ci;

 

INSERT INTO
    T_P_RED_STAND_RESULT ( ACTIVITY_NO,
    RULE_ID,
    WEIGHT,
    DEAL_TYPE,
    DATA_TYPE,
    DATA_NO,
    USER_TYPE,
    USER_NO,
    SUM_MONEY,
    SUM_NUM,
    DATE_TYPE,
    BEGIN_TIME,
    END_TIME,
    TRADE_MONEY,
    EXTEND_DATA_MEAN,
    DATA1,
    DATA2,
    DATA3,
    DATA4,
    DATA5,
    CK_REACH,
    READ_STATUS,
    PROCESS_STATUS,
    PROCESS_STATUS_DESC,
    RESULT_TYPE,
    MANAGER_NO,
    MANAGER_ORG_NO,
    CALCULATION_DATE,
    CALCULATION_TIME,
    CREATE_TIME,
    CREATE_USER_NO,
    MAINTENANCE_TIME,
    MAINTENANCE_USER_NO)  
select
    'abc' as ACTIVITY_NO,
    'def' as RULE_ID,
    1 as WEIGHT,
    'red' as DEAL_TYPE,
    'gear' as DATA_TYPE,
    '001010102' as DATA_NO,
    'merchantRed' as USER_TYPE,
    r.MERCHANT_NO as MERCHANT_NO,
    r.SUM_MONEY as SUM_MONEY,
    r.SUM_NUM as SUM_NUM ,
    r.DATE_TYPE as DATE_TYPE ,
    r.BEGIN_DATE as BEGIN_DATE,
    r.END_DATE as END_DATE,
    0 as TRADE_MONEY,
    'other' as EXTEND_DATA_MEAN,
    if(0.1 * r.SUM_MONEY >= 10 and 0.1 * r.SUM_MONEY <= 100, 0.1 * r.SUM_MONEY, 
if(0.1 * r.SUM_MONEY<10, 10, if(0.1 * r.SUM_MONEY>100, 100, 0.1 * 
r.SUM_MONEY))) as DATA1,
    '' as DATA2,
    '' as DATA3,
    '' as DATA4,
    '' as DATA5,
    '1' as CK_REACH,
    '0' as READ_STATUS,
    '' as PROCESS_STATUS,
    '' as PROCESS_STATUS_DESC,
    'flink-batch' as RESULT_TYPE,
    '' as MANAGER_NO,
    '' as MANAGER_ORG_NO,
    r.DATE_DT as CALCULATION_DATE,
    LOCALTIMESTAMP as CALCULATION_TIME,
    LOCALTIMESTAMP as CREATE_TIME,
    'system' as CREATE_USER_NO,
    '' as MAINTENANCE_TIME,
    '' as MAINTENANCE_USER_NO
from
    T_P_FILTER_MERCHANT_OTHER_DATE_RES2 r
where
    1 = 1
    and r.ACT_ID = 'abc'
    and r.RULE_ID = 'def'
    and r.DATE_DT = 20221028
    and r.MERCHANT_NO not in ( select u.USER_NO from T_P_RED_STAND_RESULT u 
where u.ACTIVITY_NO = 'abc' and u.CALCULATION_DATE = 20221028 and u.WEIGHT>=1)
    and r.MERCHANT_NO not in ( select u.USER_NO from T_P_RED_STAND_RESULT u 
where u.ACTIVITY_NO = 'abc' and u.READ_STATUS = '1')

 

We found that when executing the above flink sql, we feel that flink has loaded 
all the data in T_P_FILTER_MERCHANT_DAY_RES2 into the memory, and then 
converted the where statement into a java filter condition, and fetched the 
filtered data in the memory. We have this judgment for two reasons
1. The error message of flink is oom, memory overflow
2. The feedback from the dba said that we have done a full table query of the 
T_P_FILTER_MERCHANT_DAY_RES2 table

 

 



--
This message was sent by Atlassian Jira

[jira] [Created] (FLINK-32010) KubernetesLeaderRetrievalDriver always waits for lease update to resolve leadership

2023-05-05 Thread Jira

David Morávek created FLINK-32010:
-

 Summary: KubernetesLeaderRetrievalDriver always waits for lease 
update to resolve leadership
 Key: FLINK-32010
 URL: https://issues.apache.org/jira/browse/FLINK-32010
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.16.1, 1.17.0, 1.18.0
Reporter: David Morávek


The k8s-based leader retrieval is based on ConfigMap watching. The config map 
lifecycle (from the consumer point of view) is handled as a series of events 
with the following types:
 * ADDED -> the first time the consumer has seen the CM
 * UPDATED -> any further changes to the CM
 * DELETED -> ... you get the idea

The implementation assumes that ElectionDriver (the one that creates the CM) 
and ElectionRetriver are started simultaneously and therefore ignore the ADDED 
events because the CM is always created as empty and is updated with the 
leadership information later on.

This assumption is incorrect in the following cases (I might be missing some, 
but that's not important, the goal is to illustrate the problem):
 * TM joining the cluster later when the leaders are established to discover RM 
/ JM
 * RM tries to discover JM when 
MultipleComponentLeaderElectionDriver is used

This, for example, leads to higher job submission latencies that could be 
unnecessarily held back for up to the lease retry period [1].

[1] Configured by _high-availability.kubernetes.leader-election.retry-period_



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32009) Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeout

2023-05-05 Thread hanjie (Jira)

hanjie created FLINK-32009:
--

 Summary: Slot request bulk is not fulfillable! Could not allocate 
the required slot within slot request timeout
 Key: FLINK-32009
 URL: https://issues.apache.org/jira/browse/FLINK-32009
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN
Affects Versions: 1.14.3
Reporter: hanjie
 Attachments: jobmanager.log

Flink task lock,but yarn resource is full.
{code:java}
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: 
Slot request bulk is not fulfillable! Could not allocate the required slot 
within slot request timeout
at  
org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda$schedulePendingRequestBulkWithTimestampCheck$0(PhysicalSlotRequestBulkCheckerImpl.java:86)
 ~[flink-dist_2.11-1.14.3.jar:1.14.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[?:1.8.0_191]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_191]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRunAsync$4(AkkaRpcActor.java:455)
 ~[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
 ~[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:455)
 ~[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:213)
 ~[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at 
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
 ~[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
 ~[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at akka.actor.Actor.aroundReceive(Actor.scala:537) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at akka.actor.Actor.aroundReceive$(Actor.scala:535) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at akka.actor.ActorCell.invoke(ActorCell.scala:548) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at akka.dispatch.Mailbox.run(Mailbox.scala:231) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at akka.dispatch.Mailbox.exec(Mailbox.scala:243) 
[flink-rpc-akka_a8b5fc62-4780-45aa-879e-076def164c9f.jar:1.14.3]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_191]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) 
[?:1.8.0_191]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) 
[?:1.8.0_191]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) 
[?:1.8.0_191]
Caused by: java.util.concurrent.TimeoutException: Timeout has occurred: 30 
ms
... 29 more{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] Release flink-connector-cassandra 3.1.0, release candidate #2

2023-05-05 Thread Danny Cranmer

Hi everyone,
Please review and vote on the release candidate #2 for the version 3.1.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which are signed with the key with fingerprint
0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag v3.1.0-rc2 [5],
* website pull request listing the new release [6].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Danny

[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353030
[2]
https://dist.apache.org/repos/dist/dev/flink/flink-connector-cassandra-3.1.0-rc2
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1631
[5] https://github.com/apache/flink-connector-cassandra/tree/v3.1.0-rc2
[6] https://github.com/apache/flink-web/pull/642

Re: [VOTE] Release flink-connector-cassandra v3.1.0, release candidate #1

2023-05-05 Thread Danny Cranmer

Thanks Etienne,

To reiterate, this vote is closed. I will start the RC2.

Danny

On Thu, May 4, 2023 at 9:45 AM Etienne Chauchot 
wrote:

> @Danny, the fix is merged and the ticket is closed.
>
> You can open a RC2 whenever you want.
>
> Best
>
> Etienne
>
> Le 04/05/2023 à 06:17, Leonard Xu a écrit :
> > -1 (binding)  as the blocker issue FLINK-31927[1] found
> >
> > - Reviewed the PR [2] and left one comment that the linked issue is
> incorrect.
> >
> >
> > Best,
> > Leonard
> > [1] https://issues.apache.org/jira/browse/FLINK-31927
> > [2] https://github.com/apache/flink-connector-cassandra/pull/13
> >
> >
> >
> >> 2023年5月3日 下午5:26，Etienne Chauchot  写道：
> >>
> >> Hi all,
> >>
> >> @Danny, I just submitted the fix PR to unblock the release:
> https://github.com/apache/flink-connector-cassandra/pull/13
> >>
> >> Best
> >>
> >> Etienne
> >>
> >> Le 02/05/2023 à 14:52, Danny Cranmer a écrit :
> >>> Thanks for reporting this issue Etienne. Why was it not detected by the
> >>> unit/integration tests? Can we cover this on the CI?
> >>>
> >>> This VOTE is closed, I will open RC2 once the issue has been resolved.
> In
> >>> the meantime we could consider reopening 3.0.1 [1] for Flink 1.17
> support.
> >>> I will reopen if there is a demand for it.
> >>>
> >>> Thanks,
> >>> Danny
> >>>
> >>>
> >>> [1] https://lists.apache.org/thread/30c3yhd561o57x0prt7jqt055r4xd6lf
> >>>
> >>> On Mon, Apr 24, 2023 at 7:28 PM Etienne Chauchot  >
> >>> wrote:
> >>>
>  Hi,
> 
>  Thanks Danny for driving this new release. It now contains the new
>  source, thanks.
> 
>  I'm off but I wanted to test this release still. I made a very quick
> job
>  (1) to read from a Cassandra cluster with the new source.
> 
>  I found an issue: the source raises a j"ava.lang.NoClassDefFoundError:
>  com/codahale/metrics/Gauge" when trying to connect to the cluster on
>  Flink 1.16.0.
> 
>  As I'm on vacation right now, I don't have time to solve this now but
>  I'll do within a week.
> 
>  vote: -1 (non-binding)
> 
>  here is the blocker ticket:
>  https://issues.apache.org/jira/browse/FLINK-31927
> 
>  [1]
> 
> 
> https://github.com/echauchot/flink-samples/blob/edf4ad1624b2ad02af380efa6b5caa26bb7a274a/src/main/java/org/example/CassandraPojoSource.java
> 
>  Best
> 
>  Etienne
> 
>  Le 19/04/2023 à 21:07, Martijn Visser a écrit :
> > +1 (binding)
> >
> > - Validated hashes
> > - Verified signature
> > - Verified that no binaries exist in the source archive
> > - Build the source with Maven
> > - Verified licenses
> > - Verified web PRs
> >
> > On Fri, Apr 14, 2023 at 2:42 PM Elphas Toringepi <
> elphast...@gmail.com>
> > wrote:
> >
> >> Thanks Danny
> >>
> >> +1 (non-binding)
> >>
> >> * Checked release notes
> >> * Validated signature and checksum
> >> * Apache source builds with JDK 11
> >> * Approved website PR
> >>
> >> Kind regards,
> >> Elphas
> >>
> >>
> >> On Fri, Apr 14, 2023 at 1:14 PM Danny Cranmer <
> dannycran...@apache.org>
> >> wrote:
> >>
> >>> Hi everyone,
> >>> Please review and vote on the release candidate #1 for the version
>  3.1.0,
> >>> as follows:
> >>> [ ] +1, Approve the release
> >>> [ ] -1, Do not approve the release (please provide specific
> comments)
> >>>
> >>> This version supports both Flink 1.16.x and Flink 1.17.x
> >>>
> >>> The complete staging area is available for your review, which
> includes:
> >>> * JIRA release notes [1],
> >>> * the official Apache source release to be deployed to
> dist.apache.org
> >>> [2],
> >>> which are signed with the key with fingerprint
> >>> 0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3],
> >>> * all artifacts to be deployed to the Maven Central Repository [4],
> >>> * source code tag v3.1.0-rc1 [5],
> >>> * website pull request listing the new release [6].
> >>>
> >>> The vote will be open for at least 72 hours. It is adopted by
> majority
> >>> approval, with at least 3 PMC affirmative votes.
> >>>
> >>> Thanks,
> >>> Danny
> >>>
> >>> [1]
> >>>
> >>>
> 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353030
> >>> [2]
> >>>
> >>>
> 
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-cassandra-3.1.0-rc1/
> >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >>> [4]
> >>
> https://repository.apache.org/content/repositories/orgapacheflink-1627
> >>> [5]
> >>>
> 
> https://github.com/apache/flink-connector-cassandra/releases/tag/v3.1.0-rc1
> >>> [6] https://github.com/apache/flink-web/pull/642
> >>>
>

[ANNOUNCE] Apache flink-connector-jdbc 3.1.0 for Flink 1.16/1.17 released

2023-05-05 Thread Danny Cranmer

The Apache Flink community is very happy to announce the release of Apache
flink-connector-jdbc 3.1.0. This connector supports Flink 1.16 and 1.17.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352678

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Danny

Re: [DISCUSS] FLIP-306: Unified File Merging Mechanism for Checkpoints

2023-05-05 Thread Zakelly Lan

Hi everyone,

It seems there are no more questions unresolved. So I would like to
start a vote on May 8th at 10:00 AM GMT.

Please let me know if you have any concerns, thanks!


Best,
Zakelly

On Fri, May 5, 2023 at 4:03 PM Rui Fan <1996fan...@gmail.com> wrote:
>
> Hi Zakelly,
>
> Thanks for the clarification!
>
> Currently, I understand what you mean, and LGTM.
>
> Best,
> Rui Fan
>
> On Fri, May 5, 2023 at 12:27 PM Zakelly Lan  wrote:
>
> > Hi all,
> >
> > @Yun Tang and I have an offline discussion, and we agreed that:
> >
> > 1. The design of this FLIP is pretty much like the option 3 in design
> > doc[1] for FLINK-23342, and it is almost the best solution in general.
> > Based on our production experience, this FLIP can solve the file flood
> > problem very well.
> > 2. There is a corner case that the directory may be left over when the
> > job stops, so I added some content in section 4.8.
> >
> > Best,
> > Zakelly
> >
> >
> > [1]
> > https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit#
> >
> > On Fri, May 5, 2023 at 11:19 AM Zakelly Lan  wrote:
> > >
> > > Hi Rui Fan,
> > >
> > > Thanks for your reply.
> > >
> > > > Maybe re-uploaded in the next checkpoint is also a general solution
> > > > for shared state? If yes, could we consider it as an optimization?
> > > > And we can do it after the FLIP is done.
> > >
> > > Yes, it is a general solution for shared states. Maybe in the first
> > > version we can let the shared states not re-use any previous state
> > > handle after restoring, thus the state backend will do a full snapshot
> > > and re-uploading the files it needs. This could cover the scenario
> > > that rocksdb only uploads the base DB files. And later we could
> > > consider performing fast copy in DFS to optimize the re-uploading.
> > > WDYT?
> > >
> > >
> > > > I'm not sure why we need 2 configurations, or whether 1 configuration
> > > > is enough here.
> > >
> > > > The `max-file-pool-size` is hard to give a default value. For jobs with
> > > > many tasks in a TM, it may be useful for file merging. However,
> > > > it doesn't work well for jobs with a small number of tasks in a TM.
> > >
> > > > I prefer just adding the `max-file-pool-size`, and
> > > > the `pool size = number of tasks / max-file-pool-size`. WDYT?
> > >
> > >
> > > Sorry for not explaining clearly. The value of pool size is calculated
> > by:
> > >
> > > 1. pool size = number of tasks / max-subtasks-per-file
> > > 2. if pool size > max-file-pool-size then pool size = max-file-pool-size
> > >
> > > The `max-subtasks-per-file` addresses the issue of sequential file
> > > writing, while the `max-file-pool-size` acts as a safeguard to prevent
> > > an excessively large file pool. WDYT?
> > >
> > >
> > > Thanks again for your thoughts.
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Thu, May 4, 2023 at 3:52 PM Rui Fan <1996fan...@gmail.com> wrote:
> > > >
> > > > Hi Zakelly,
> > > >
> > > > Sorry for the late reply, I still have some minor questions.
> > > >
> > > > >> (3) When rescaling, do all shared files need to be copied?
> > > > >
> > > > > I agree with you that only sst files of the base DB need to be copied
> > > > > (or re-uploaded in the next checkpoint). However, section 4.2
> > > > > simplifies file copying issues (copying all files), following the
> > > > > concept of shared state.
> > > >
> > > > Maybe re-uploaded in the next checkpoint is also a general solution
> > > > for shared state? If yes, could we consider it as an optimization?
> > > > And we can do it after the FLIP is done.
> > > >
> > > > >> (5) How many physical files can a TM write at the same checkpoint
> > at the
> > > > same time?
> > > > >
> > > > > This is a very good point. Actually, there is a file reuse pool as
> > > > > section 4.6 described. There could be multiple files within this
> > pool,
> > > > > supporting concurrent writing by multiple writers. I suggest
> > providing
> > > > > two configurations to control the file number:
> > > > >
> > > > >   state.checkpoints.file-merging.max-file-pool-size: Specifies the
> > > > > upper limit of the file pool size.
> > > > >   state.checkpoints.file-merging.max-subtasks-per-file: Specifies the
> > > > > lower limit of the file pool size based on the number of subtasks
> > > > > within each TM.
> > > > >
> > > > > The number of simultaneously open files is controlled by these two
> > > > > options, and the first option takes precedence over the second.
> > > >
> > > > I'm not sure why we need 2 configurations, or whether 1 configuration
> > > > is enough here.
> > > >
> > > > The `max-file-pool-size` is hard to give a default value. For jobs with
> > > > many tasks in a TM, it may be useful for file merging. However,
> > > > it doesn't work well for jobs with a small number of tasks in a TM.
> > > >
> > > > I prefer just adding the `max-file-pool-size`, and
> > > > the `pool size = number of tasks / max-file-pool-size`. WDYT?
> > > >
> > > > Maybe I missed

[RESULT] [VOTE] flink-connector-jdbc 3.1.0, release candidate #1

2023-05-05 Thread Danny Cranmer

I'm happy to announce that we have unanimously approved this release.

There are 6 approving votes, 3 of which are binding:
* Elphas Toringepi
* Hong Teoh
* Benchao Li
* Danny Cranmer (binding)
* Martijn Visser (binding)
* Leonard Xu (binding)

There are no disapproving votes.

Thanks everyone!
Danny

Re: [VOTE] Release flink-connector-jdbc v3.1.0, release candidate #1

2023-05-05 Thread Danny Cranmer

Thanks all, this vote is now closed. I will announce the results in a
separate thread.

Danny

On Thu, May 4, 2023 at 4:54 AM Leonard Xu  wrote:

>  +1 (binding)
>
> - built from source code succeeded
> - verified signatures
> - verified hashsums
> - checked Github release tag
> - checked release notes
> - reviewed the web PR
> - found the JDBC docs contains a valid link, reopened FLINK-31759[1] to
> fix it, not a blocker
>
> Best,
> Leonard
> [1]https://issues.apache.org/jira/browse/FLINK-31759 <
> https://issues.apache.org/jira/browse/FLINK-31759>
>
> > 2023年5月2日 下午8:25，Danny Cranmer  写道：
> >
> > +1 (binding)
> >
> > - Release notes look good
> > - Source archive signature and checksums good
> > - Verified there are no binaries in the source archive
> > - Contents of Maven repo look good
> > - Built from source/tests pass
> > - Tag is present in Github
> > - Fixed web PR as per feedback
> >
> > Thanks,
> > Danny
> >
> > On Wed, Apr 19, 2023 at 8:22 PM Martijn Visser  >
> > wrote:
> >
> >> +1 (binding)
> >>
> >> - Validated hashes
> >> - Verified signature
> >> - Verified that no binaries exist in the source archive
> >> - Build the source with Maven
> >> - Verified licenses
> >> - Verified web PRs
> >>
> >> On Wed, Apr 19, 2023 at 9:35 AM Martijn Visser <
> martijnvis...@apache.org>
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> There was already a blocker ticken op for this, that's
> >>> https://issues.apache.org/jira/browse/FLINK-31770. There's also an
> open
> >>> PR that should resolve it, but is waiting on a reviewer. See
> >>> https://github.com/apache/flink-connector-jdbc/pull/22
> >>>
> >>> I've already done the administration, so this is just fyi :)
> >>>
> >>> Best regards,
> >>>
> >>> Martijn
> >>>
> >>> On Wed, Apr 19, 2023 at 9:17 AM Danny Cranmer  >
> >>> wrote:
> >>>
>  Thank Benchao for the hotfix. I noticed the CI is also failing/flaky
> for
>  the main branch, I have raised a follow-up [1].
> 
>  Danny,
> 
>  [1] https://issues.apache.org/jira/browse/FLINK-31847
> 
>  On Sun, Apr 16, 2023 at 10:57 AM Benchao Li 
> >> wrote:
> 
> > And I've filed a hotfix PR[1] to fix the copyright issue.
> >
> > [1] https://github.com/apache/flink-connector-jdbc/pull/41
> >
> > Benchao Li  于2023年4月16日周日 17:51写道：
> >
> >> * Verified hashes, signatures (OK)
> >> * Download, build and test source code (OK)
> >> * Checked files in Nexus staging (OK)
> >> * Start a Flink local cluster (1.17.0), run sql-client with jdbc
> >> connector jar with Mysql 8.0.28 (OK)
> >> * Go through the release note (OK)
> >>
> >> One thing I spotted is that when I checked the NOTICE file, I found
>  that
> >> we have not updated the copyright year to 2023, I'm not sure whether
>  we
> >> should take this as a blocker issue. If not, my vote is +1.
> >>
> >> And I left my comments in the PR for website updating.
> >>
> >> Teoh, Hong  于2023年4月14日周五 15:54写道：
> >>
> >>> Thanks Danny
> >>>
> >>> +1 (non-binding)
> >>>
> >>> * Hashes and Signatures look good
> >>> * All required files on dist.apache.org
> >>> * Tag is present in Github
> >>> * Verified source archive does not contain any binary files
> >>> * Source archive builds using maven
> >>>
> >>>
> >>> Cheers,
> >>> Hong
> >>>
> >>>
>  On 14 Apr 2023, at 05:21, Elphas Toringepi  >>>
> >>> wrote:
> 
>  CAUTION: This email originated from outside of the organization.
> >> Do
> > not
> >>> click links or open attachments unless you can confirm the sender
> >> and
> > know
> >>> the content is safe.
> 
> 
> 
>  +1 (non-binding)
> 
>  * JIRA release notes look good
>  * Verified signature and checksum for Apache source
>  * Checked source code tag and confirmed Github and Apache source
> >>> releases
>  are identical
>  * Reviewed website pull request and left comment
> 
>  Kind regards,
>  Elphas
> 
> 
>  On Thu, Apr 13, 2023 at 3:42 PM Danny Cranmer <
> > dannycran...@apache.org>
>  wrote:
> 
> > Hi everyone,
> > Please review and vote on the release candidate #1 for the
> >> version
> >>> v3.1.0,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific
>  comments)
> >
> > This version supports both Flink 1.16.x and Flink 1.17.x
> >
> > The complete staging area is available for your review, which
> > includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to
> > dist.apache.org
> > [2],
> > which are signed with the key with fingerprint
> > 0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3],
>

Re: [DISCUSS] FLIP-306: Unified File Merging Mechanism for Checkpoints

2023-05-05 Thread Rui Fan

Hi Zakelly,

Thanks for the clarification!

Currently, I understand what you mean, and LGTM.

Best,
Rui Fan

On Fri, May 5, 2023 at 12:27 PM Zakelly Lan  wrote:

> Hi all,
>
> @Yun Tang and I have an offline discussion, and we agreed that:
>
> 1. The design of this FLIP is pretty much like the option 3 in design
> doc[1] for FLINK-23342, and it is almost the best solution in general.
> Based on our production experience, this FLIP can solve the file flood
> problem very well.
> 2. There is a corner case that the directory may be left over when the
> job stops, so I added some content in section 4.8.
>
> Best,
> Zakelly
>
>
> [1]
> https://docs.google.com/document/d/1NJJQ30P27BmUvD7oa4FChvkYxMEgjRPTVdO1dHLl_9I/edit#
>
> On Fri, May 5, 2023 at 11:19 AM Zakelly Lan  wrote:
> >
> > Hi Rui Fan,
> >
> > Thanks for your reply.
> >
> > > Maybe re-uploaded in the next checkpoint is also a general solution
> > > for shared state? If yes, could we consider it as an optimization?
> > > And we can do it after the FLIP is done.
> >
> > Yes, it is a general solution for shared states. Maybe in the first
> > version we can let the shared states not re-use any previous state
> > handle after restoring, thus the state backend will do a full snapshot
> > and re-uploading the files it needs. This could cover the scenario
> > that rocksdb only uploads the base DB files. And later we could
> > consider performing fast copy in DFS to optimize the re-uploading.
> > WDYT?
> >
> >
> > > I'm not sure why we need 2 configurations, or whether 1 configuration
> > > is enough here.
> >
> > > The `max-file-pool-size` is hard to give a default value. For jobs with
> > > many tasks in a TM, it may be useful for file merging. However,
> > > it doesn't work well for jobs with a small number of tasks in a TM.
> >
> > > I prefer just adding the `max-file-pool-size`, and
> > > the `pool size = number of tasks / max-file-pool-size`. WDYT?
> >
> >
> > Sorry for not explaining clearly. The value of pool size is calculated
> by:
> >
> > 1. pool size = number of tasks / max-subtasks-per-file
> > 2. if pool size > max-file-pool-size then pool size = max-file-pool-size
> >
> > The `max-subtasks-per-file` addresses the issue of sequential file
> > writing, while the `max-file-pool-size` acts as a safeguard to prevent
> > an excessively large file pool. WDYT?
> >
> >
> > Thanks again for your thoughts.
> >
> > Best,
> > Zakelly
> >
> > On Thu, May 4, 2023 at 3:52 PM Rui Fan <1996fan...@gmail.com> wrote:
> > >
> > > Hi Zakelly,
> > >
> > > Sorry for the late reply, I still have some minor questions.
> > >
> > > >> (3) When rescaling, do all shared files need to be copied?
> > > >
> > > > I agree with you that only sst files of the base DB need to be copied
> > > > (or re-uploaded in the next checkpoint). However, section 4.2
> > > > simplifies file copying issues (copying all files), following the
> > > > concept of shared state.
> > >
> > > Maybe re-uploaded in the next checkpoint is also a general solution
> > > for shared state? If yes, could we consider it as an optimization?
> > > And we can do it after the FLIP is done.
> > >
> > > >> (5) How many physical files can a TM write at the same checkpoint
> at the
> > > same time?
> > > >
> > > > This is a very good point. Actually, there is a file reuse pool as
> > > > section 4.6 described. There could be multiple files within this
> pool,
> > > > supporting concurrent writing by multiple writers. I suggest
> providing
> > > > two configurations to control the file number:
> > > >
> > > >   state.checkpoints.file-merging.max-file-pool-size: Specifies the
> > > > upper limit of the file pool size.
> > > >   state.checkpoints.file-merging.max-subtasks-per-file: Specifies the
> > > > lower limit of the file pool size based on the number of subtasks
> > > > within each TM.
> > > >
> > > > The number of simultaneously open files is controlled by these two
> > > > options, and the first option takes precedence over the second.
> > >
> > > I'm not sure why we need 2 configurations, or whether 1 configuration
> > > is enough here.
> > >
> > > The `max-file-pool-size` is hard to give a default value. For jobs with
> > > many tasks in a TM, it may be useful for file merging. However,
> > > it doesn't work well for jobs with a small number of tasks in a TM.
> > >
> > > I prefer just adding the `max-file-pool-size`, and
> > > the `pool size = number of tasks / max-file-pool-size`. WDYT?
> > >
> > > Maybe I missed some information. Please correct me if I'm wrong,
> thanks.
> > >
> > > Best,
> > > Rui Fan
> > >
> > > On Fri, Apr 28, 2023 at 12:10 AM Zakelly Lan 
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Thanks for all the feedback so far.
> > > >
> > > > The discussion has been going on for some time, and all the comments
> > > > and suggestions are addressed. So I would like to start a vote on
> this
> > > > FLIP, which begins a week later (May. 5th at 10:00 AM GMT).
> > > >
> > > > If you have any concerns,

[DISCUSS] FLIP-310:use VARINT and ZIGZAG to encode ROWDATA in state

2023-05-05 Thread Xiaogang Zhou

Hi Guys,

I have created a FLIP WIKI FLIP-310
,
and documented my thinking about using the varint format in FLINK state to
improve the FLINK state performance.

Would you please help review and let me know what you think?

Re: [VOTE] Release flink-connector-gcp-pubsub v3.0.1, release candidate #1

2023-05-05 Thread Khanh Vu

+1 (non-binding)

- Verified sha512 checksum matches file archive.
- Verified file archive is signed and signature is authorized.
- Verified no binaries exist in the source archive.
- Verified source archive is consistent with Github source code with
tag v3.0.1-rc1, at commit 73e56edb2aa4513f6a73dc071545fb2508fd2d44
- Source built successfully with maven.
- Executed end to end tests successfully for flink versions: 1.15.4,
1.16.1, 1.17.0

Best regards,
Khanh Vu


On Thu, May 4, 2023 at 5:01 AM Leonard Xu  wrote:

>  +1 (binding)
>
> - built from source code succeeded
> - verified signatures
> - verified hashsums
> - checked Github release tag
> - checked release notes
> - reviewed the web PR
>
> Best,
> Leonard
>
> > 2023年4月20日 上午3:17，Martijn Visser  写道：
> >
> > +1 (binding)
> >
> > - Validated hashes
> > - Verified signature
> > - Verified that no binaries exist in the source archive
> > - Build the source with Maven
> > - Verified licenses
> > - Verified web PRs
> >
> > On Fri, Apr 14, 2023 at 6:15 PM Ryan Skraba  >
> > wrote:
> >
> >> Hello!  +1 (non-binding)
> >>
> >> I've validated the source for the RC1:
> >> flink-connector-gcp-pubsub-3.0.1-src.tgz
> >> * The sha512 checksum is OK.
> >> * The source file is signed correctly.
> >> * The signature A5F3BCE4CBE993573EC5966A65321B8382B219AF is found in the
> >> KEYS file, and on https://keys.openpgp.org
> >> * The source file is consistent with the Github tag v3.0.1-rc1, which
> >> corresponds to commit 73e56edb2aa4513f6a73dc071545fb2508fd2d44
> >>   - The files explicitly excluded by create_pristine_sources (such as
> >> .gitignore and the submodule tools/releasing/shared) are not present.
> >> * Has a LICENSE file and a NOTICE filel
> >> * Does not contain any compiled binaries.
> >>
> >> * The sources can be compiled and unit tests pass with flink.version
> 1.17.0
> >> and flink.version 1.16.1
> >>
> >> * Nexus has three staged artifact ids for 3.0.1-1.16 and 3.0.1-1.17
> >> - flink-connector-gcp-pubsub (.jar, -javadoc.jar, -sources.jar and .pom)
> >> - flink-connector-gcp-pubsub-e2e-tests (.jar, -sources.jar and .pom)
> >> - flink-connector-gcp-pubsub-parent (only .pom)
> >> * All 16 files have been signed with the same key as above, and have
> >> correct sha1 and md5 checksums.
> >>
> >> Simple smoke testing on an emulated Pub/Sub service works for both flink
> >> versions.
> >>
> >> One really minor point: it looks like the
> >> org.apache.flink:flink-connector-gcp-pubsub-parent:3.0.1-1.17:pom has
> the
> >> flink-version set to 1.16.0.  This is a bit confusing, but all the flink
> >> transitive dependencies are in the provided scope, so there's no
> >> consequence.  I guess we could argue that it is the "source"
> compatibility
> >> level for both versions!
> >>
> >> All my best and thanks for the release.
> >>
> >> Ryan
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Apr 13, 2023 at 4:07 PM Martijn Visser <
> martijnvis...@apache.org>
> >> wrote:
> >>
> >>> Hi everyone,
> >>> Please review and vote on the release candidate #1 for the version
> 3.0.1,
> >>> as follows:
> >>> [ ] +1, Approve the release
> >>> [ ] -1, Do not approve the release (please provide specific comments)
> >>>
> >>> This version is compatible with Flink 1.16.x and Flink 1.17.x.
> >>>
> >>> The complete staging area is available for your review, which includes:
> >>> * JIRA release notes [1],
> >>> * the official Apache source release to be deployed to dist.apache.org
> >>> [2],
> >>> which are signed with the key with fingerprint
> >>> A5F3BCE4CBE993573EC5966A65321B8382B219AF [3],
> >>> * all artifacts to be deployed to the Maven Central Repository [4],
> >>> * source code tag v3.0.1-rc1 [5],
> >>> * website pull request listing the new release [6].
> >>>
> >>> The vote will be open for at least 72 hours. It is adopted by majority
> >>> approval, with at least 3 PMC affirmative votes.
> >>>
> >>> Thanks,
> >>> Release Manager
> >>>
> >>> [1]
> >>>
> >>>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352770
> >>> [2]
> >>>
> >>>
> >>
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-gcp-pubsub-3.0.1-rc1
> >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >>> [4]
> >>>
> https://repository.apache.org/content/repositories/orgapacheflink-1613/
> >>> [5]
> >>>
> >>>
> >>
> https://github.com/apache/flink-connector-gcp-pubsub/releases/tag/v3.0.1-rc1
> >>> [6] https://github.com/apache/flink-web/pull/637
> >>>
> >>
>
>

Re: [VOTE] Release flink-connector-rabbitmq v3.0.1, release candidate #1

2023-05-05 Thread Khanh Vu

Sorry, the above report is supposed for flink-connector-gpc-pubsub-3.0.1

-
Here is the check for this one:

+1 (non-binding)

- Verified sha512 checksum matches file archive.
- Verified file archive is signed and signature is authenticated.
- Verified no binaries exist in the source archive.
- Verified source archive is consistent with Github source code with tag
v3.0.1-rc1, at commit 9827e71662c8f155cda5efe5ebbac804fd0fd8e2
- Source built successfully with maven.
- (No end to end tests run for this connector)

Best regards,
Khanh Vu


On Fri, May 5, 2023 at 7:55 AM Khanh Vu  wrote:

> +1 (non-binding)
>
> - Verified sha512 checksum matches file archive.
> - Verified file archive is signed and signature is authorized.
> - Verified no binaries exist in the source archive.
> - Verified source archive is consistent with Github source code with
> tag v3.0.1-rc1, at commit 73e56edb2aa4513f6a73dc071545fb2508fd2d44
> - Source built successfully with maven.
> - Executed end to end tests successfully for flink versions: 1.15.4,
> 1.16.1, 1.17.0
>
> Best regards,
> Khanh Vu
>
>
> On Thu, May 4, 2023 at 3:47 AM Leonard Xu  wrote:
>
>>
>> +1 (binding)
>>
>> - built from source code succeeded
>> - verified signatures
>> - verified hashsums
>> - checked Github release tag
>> - reviewed the web PR
>>
>> Best,
>> Leonard
>>
>>
>> 2023年4月20日 上午3:29，Martijn Visser  写道：
>>
>> +1 (binding)
>>
>> - Validated hashes
>> - Verified signature
>> - Verified that no binaries exist in the source archive
>> - Build the source with Maven
>> - Verified licenses
>> - Verified web PRs
>>
>> On Mon, Apr 17, 2023 at 7:00 PM Ryan Skraba > >
>> wrote:
>>
>> Hello!  +1 (non-binding)
>>
>> I've validated the source for the RC1:
>> flink-connector-rabbitmq-3.0.1-src.tgz
>> * The sha512 checksum is OK.
>> * The source file is signed correctly.
>> * The signature A5F3BCE4CBE993573EC5966A65321B8382B219AF is found in the
>> KEYS file, and on https://keys.openpgp.org
>> * The source file is consistent with the Github tag v3.0.1-rc1, which
>> corresponds to commit 9827e71662c8f155cda5efe5ebbac804fd0fd8e2
>>   - The files explicitly excluded by create_pristine_sources (such as
>> .gitignore and the submodule tools/releasing/shared) are not present.
>> * Has a LICENSE file and a NOTICE file.  The sql-connector has a
>> NOTICE file for bundled artifacts.
>> * Does not contain any compiled binaries.
>>
>> * The sources can be compiled and tests pass with flink.version 1.17.0 and
>> flink.version 1.16.1
>>
>> * Nexus has three staged artifact ids for 3.0.1-1.16 and 3.0.1-1.17
>> - flink-connector-rabbitmq-parent (only .pom)
>> - flink-connector-rabbitmq (.jar, -sources.jar, -javadoc.jar and .pom)
>> - flink-sql-connector-rabbitmq (.jar, -sources.jar and .pom)
>> * All 16 files have been signed with the same key as above, and have
>> correct sha1 and md5 checksums.
>>
>> I didn't run any additional smoke tests other than the integration test
>> cases.
>>
>> A couple minor points, but nothing that would block this release.
>>
>> - like flink-connector-gcp-pubsub-parent, the
>> flink-connector-rabbitmq-parent:3.0.1-1.17 pom artifact has the
>> flink.version set to 1.16.0, which might be confusing.
>> - the NOTICE file for sql-connector has the wrong year.
>>
>> All my best and thanks for the release.
>>
>> Ryan
>>
>>
>> On Thu, Apr 13, 2023 at 4:45 PM Martijn Visser 
>> wrote:
>>
>> Hi everyone,
>> Please review and vote on the release candidate #1 for the version 3.0.1,
>> as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>> This version is compatible with Flink 1.16.x and Flink 1.17.x
>>
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * the official Apache source release to be deployed to dist.apache.org
>> [2],
>> which are signed with the key with fingerprint
>> A5F3BCE4CBE993573EC5966A65321B8382B219AF [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag v3.0.1-rc1 [5],
>> * website pull request listing the new release [6].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> Thanks,
>> Release Manager
>>
>> [1]
>>
>>
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352699
>>
>> [2]
>>
>>
>>
>> https://dist.apache.org/repos/dist/dev/flink/flink-connector-rabbitmq-3.0.1-rc1
>>
>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> [4]
>> https://repository.apache.org/content/repositories/orgapacheflink-1615/
>> [5]
>>
>> https://github.com/apache/flink-connector-rabbitmq/releases/tag/v3.0.1-rc1
>>
>> [6] https://github.com/apache/flink-web/pull/639
>>
>>
>>
>>

Re: [VOTE] Release flink-connector-rabbitmq v3.0.1, release candidate #1

2023-05-05 Thread Khanh Vu

+1 (non-binding)

- Verified sha512 checksum matches file archive.
- Verified file archive is signed and signature is authorized.
- Verified no binaries exist in the source archive.
- Verified source archive is consistent with Github source code with
tag v3.0.1-rc1, at commit 73e56edb2aa4513f6a73dc071545fb2508fd2d44
- Source built successfully with maven.
- Executed end to end tests successfully for flink versions: 1.15.4,
1.16.1, 1.17.0

Best regards,
Khanh Vu


On Thu, May 4, 2023 at 3:47 AM Leonard Xu  wrote:

>
> +1 (binding)
>
> - built from source code succeeded
> - verified signatures
> - verified hashsums
> - checked Github release tag
> - reviewed the web PR
>
> Best,
> Leonard
>
>
> 2023年4月20日 上午3:29，Martijn Visser  写道：
>
> +1 (binding)
>
> - Validated hashes
> - Verified signature
> - Verified that no binaries exist in the source archive
> - Build the source with Maven
> - Verified licenses
> - Verified web PRs
>
> On Mon, Apr 17, 2023 at 7:00 PM Ryan Skraba 
> wrote:
>
> Hello!  +1 (non-binding)
>
> I've validated the source for the RC1:
> flink-connector-rabbitmq-3.0.1-src.tgz
> * The sha512 checksum is OK.
> * The source file is signed correctly.
> * The signature A5F3BCE4CBE993573EC5966A65321B8382B219AF is found in the
> KEYS file, and on https://keys.openpgp.org
> * The source file is consistent with the Github tag v3.0.1-rc1, which
> corresponds to commit 9827e71662c8f155cda5efe5ebbac804fd0fd8e2
>   - The files explicitly excluded by create_pristine_sources (such as
> .gitignore and the submodule tools/releasing/shared) are not present.
> * Has a LICENSE file and a NOTICE file.  The sql-connector has a
> NOTICE file for bundled artifacts.
> * Does not contain any compiled binaries.
>
> * The sources can be compiled and tests pass with flink.version 1.17.0 and
> flink.version 1.16.1
>
> * Nexus has three staged artifact ids for 3.0.1-1.16 and 3.0.1-1.17
> - flink-connector-rabbitmq-parent (only .pom)
> - flink-connector-rabbitmq (.jar, -sources.jar, -javadoc.jar and .pom)
> - flink-sql-connector-rabbitmq (.jar, -sources.jar and .pom)
> * All 16 files have been signed with the same key as above, and have
> correct sha1 and md5 checksums.
>
> I didn't run any additional smoke tests other than the integration test
> cases.
>
> A couple minor points, but nothing that would block this release.
>
> - like flink-connector-gcp-pubsub-parent, the
> flink-connector-rabbitmq-parent:3.0.1-1.17 pom artifact has the
> flink.version set to 1.16.0, which might be confusing.
> - the NOTICE file for sql-connector has the wrong year.
>
> All my best and thanks for the release.
>
> Ryan
>
>
> On Thu, Apr 13, 2023 at 4:45 PM Martijn Visser 
> wrote:
>
> Hi everyone,
> Please review and vote on the release candidate #1 for the version 3.0.1,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> This version is compatible with Flink 1.16.x and Flink 1.17.x
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2],
> which are signed with the key with fingerprint
> A5F3BCE4CBE993573EC5966A65321B8382B219AF [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag v3.0.1-rc1 [5],
> * website pull request listing the new release [6].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1]
>
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352699
>
> [2]
>
>
>
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-rabbitmq-3.0.1-rc1
>
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1615/
> [5]
>
> https://github.com/apache/flink-connector-rabbitmq/releases/tag/v3.0.1-rc1
>
> [6] https://github.com/apache/flink-web/pull/639
>
>
>
>

Re: 您好，退订

2023-05-05 Thread Leonard Xu

Please send email to dev-unsubscr...@flink.apache.org if you want to 
unsubscribe the mail from dev@flink.apache.org, and you can refer[1][2] for 
more details.
请发送任意内容的邮件到 dev-unsubscr...@flink.apache.org 地址来取消订阅来自 dev@flink.apache.org 
邮件组的邮件，你可以参考[1][2] 管理邮件订阅。

Best,
Leonard

[1]https://flink.apache.org/zh/community/#%e9%82%ae%e4%bb%b6%e5%88%97%e8%a1%a8
[2]https://flink.apache.org/community.html#mailing-lists

[jira] [Created] (FLINK-32008) Protobuf format throws exception with Map datatype

2023-05-05 Thread Xuannan Su (Jira)

Xuannan Su created FLINK-32008:
--

 Summary: Protobuf format throws exception with Map datatype
 Key: FLINK-32008
 URL: https://issues.apache.org/jira/browse/FLINK-32008
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.17.0
Reporter: Xuannan Su
 Attachments: flink-protobuf-example.zip

The protobuf format throws exception when working with Map data type. I 
uploaded a example project to reproduce the problem.

 
{code:java}
Caused by: java.lang.RuntimeException: One or more fetchers have encountered 
exception
    at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:261)
    at 
org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
    at 
org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:131)
    at 
org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:417)
    at 
org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
    at 
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
    at 
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:550)
    at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
    at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:839)
    at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:788)
    at 
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
    at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:931)
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: SplitFetcher thread 0 received 
unexpected exception while polling the records
    at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:165)
    at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:114)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    ... 1 more
Caused by: java.io.IOException: Failed to deserialize PB object.
    at 
org.apache.flink.formats.protobuf.deserialize.PbRowDataDeserializationSchema.deserialize(PbRowDataDeserializationSchema.java:75)
    at 
org.apache.flink.formats.protobuf.deserialize.PbRowDataDeserializationSchema.deserialize(PbRowDataDeserializationSchema.java:42)
    at 
org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:82)
    at 
org.apache.flink.connector.file.table.DeserializationSchemaAdapter$LineBytesInputFormat.readRecord(DeserializationSchemaAdapter.java:197)
    at 
org.apache.flink.connector.file.table.DeserializationSchemaAdapter$LineBytesInputFormat.nextRecord(DeserializationSchemaAdapter.java:210)
    at 
org.apache.flink.connector.file.table.DeserializationSchemaAdapter$Reader.readBatch(DeserializationSchemaAdapter.java:124)
    at 
org.apache.flink.connector.file.src.util.RecordMapperWrapperRecordIterator$1.readBatch(RecordMapperWrapperRecordIterator.java:82)
    at 
org.apache.flink.connector.file.src.impl.FileSourceSplitReader.fetch(FileSourceSplitReader.java:67)
    at 
org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58)
    at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:162)
    ... 6 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at 
org.apache.flink.formats.protobuf.deserialize.ProtoToRowConverter.convertProtoBinaryToRow(ProtoToRowConverter.java:129)
    at 
org.apache.flink.formats.protobuf.deserialize.PbRowDataDeserializationSchema.deserialize(PbRowDataDeserializationSchema.java:70)
    ... 15 more
Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a 
protocol message, the input ended unexpectedly in the middle of a field.  This 
could mean either that the input has been truncated or that an embedded

Re: 您好，退订

2023-05-05 Thread Biao Geng

你好，可以通过发送包含任意内容的邮件到  dev-unsubscr...@flink.apache.org 来取消dev订阅。
Best,
Biao Geng

Fang jun  于2023年5月5日周五 14:07写道：

>
>
> Sent from my iPhone
>

Re: 退订

2023-05-05 Thread Biao Geng

你好，可以通过发送包含任意内容的邮件到  dev-unsubscr...@flink.apache.org 来取消dev订阅。
Best,
Biao Geng

houyui <2008-ho...@163.com> 于2023年5月5日周五 13:33写道：

> 您好：
>退订！

[jira] [Created] (FLINK-32022) Source level scaling is not applied to operators with chaining applied

[jira] [Created] (FLINK-32021) Improvement the Javadoc for SpecifiedOffsetsInitializer and TimestampOffsetsInitializer.

[jira] [Created] (FLINK-32020) Enable Dynamic Partition Discovery by Default in Kafka Source based on FLIP-288

[jira] [Created] (FLINK-32019) EARLIEST offset strategy for partitions discoveried later based on FLIP-288

[jira] [Created] (FLINK-32018) Many builds of benchmark have been interrupted since 20230428

[jira] [Created] (FLINK-32017) DISTINCT COUNT result is incorrect with ttl

[jira] [Created] (FLINK-32016) Logs are printed as disabled even if the autoscaler option is true

[jira] [Created] (FLINK-32015) Flink does not fail SSL Handshake when expired cert is used

[jira] [Created] (FLINK-32014) Cassandra source documentation is missing

[jira] [Created] (FLINK-32013) Moves DefaultLeaderElectionService.startLeaderElectionBackend() into HAServices

[jira] [Created] (FLINK-32012) Operator failed to rollback due to missing HA metadata

[jira] [Created] (FLINK-32011) flink1.15.2 loaded all the data in the table in mysql5.7

[jira] [Created] (FLINK-32010) KubernetesLeaderRetrievalDriver always waits for lease update to resolve leadership

[jira] [Created] (FLINK-32009) Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeout

[VOTE] Release flink-connector-cassandra 3.1.0, release candidate #2

Re: [VOTE] Release flink-connector-cassandra v3.1.0, release candidate #1

[ANNOUNCE] Apache flink-connector-jdbc 3.1.0 for Flink 1.16/1.17 released

Re: [DISCUSS] FLIP-306: Unified File Merging Mechanism for Checkpoints

[RESULT] [VOTE] flink-connector-jdbc 3.1.0, release candidate #1

Re: [VOTE] Release flink-connector-jdbc v3.1.0, release candidate #1

Re: [DISCUSS] FLIP-306: Unified File Merging Mechanism for Checkpoints

[DISCUSS] FLIP-310:use VARINT and ZIGZAG to encode ROWDATA in state

Re: [VOTE] Release flink-connector-gcp-pubsub v3.0.1, release candidate #1

Re: [VOTE] Release flink-connector-rabbitmq v3.0.1, release candidate #1

Re: [VOTE] Release flink-connector-rabbitmq v3.0.1, release candidate #1

Re: 您好，退订

[jira] [Created] (FLINK-32008) Protobuf format throws exception with Map datatype

Re: 您好，退订

Re: 退订

29 matches

Site Navigation

Mail list logo

Footer information