[ 
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huajiewang updated FLINK-20972:
-------------------------------
    Description: 
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,

Maybe A large number of EventData will be output (log.info)

,which will cause IO bottleneck and disk waste

 
 my code in the attachment, A large number event data output in the log output 
by flink, 
  
 like: 
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction 
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b, 
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ), 
transactionStartTime=1610426158532} from checkpoint 4{code}
 method notifyCheckpointComplete of TwoPhaseCommitSinkFunction,  about LOG.info 
code is as follows:
{code:java}
  LOG.info("{} - checkpoint {} complete, committing transaction {} from 
checkpoint {}", name(), checkpointId, pendingTransaction, 
pendingTransactionCheckpointId);  
 {code}
will be call the toString method of pendingTransaction (TransactionHolder), 
TransactionHolder'toString method code is:

 


 @Override
 public String toString() {
 return "TransactionHolder

{" + "handle=" + handle + ", transactionStartTime=" + transactionStartTime + '}

';
 }
  
 handle is the concrete realization of my Transaction! There is a parameter of 
List type in my Transaction, which is used to receive data. as a result, these 
data are printed out(log.info)
  
  

 

  was:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,

Maybe A large number of EventData will be output (log.info)

,which will cause IO bottleneck and disk waste

 
 my code in the attachment, A large number event data output in the log output 
by flink, 
  
 like: 


 Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction 
TransactionHolde \{handle=Transaction(b420c880a951403984f231dd7e33597b, 
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ), 
transactionStartTime=1610426158532} from checkpoint 4

 

method notifyCheckpointComplete of TwoPhaseCommitSinkFunction,  about LOG.info 
code is as follows:

LOG.info("{} - checkpoint {} complete, committing transaction {} from 
checkpoint {}",
 name(), checkpointId, pendingTransaction, pendingTransactionCheckpointId);

 
 will be call the toString method of pendingTransaction (TransactionHolder), 
TransactionHolder'toString method code is:
 @Override
 public String toString() {
 return "TransactionHolder

{" + "handle=" + handle + ", transactionStartTime=" + transactionStartTime + '}

';
 }
  
 handle is the concrete realization of my Transaction! There is a parameter of 
List type in my Transaction, which is used to receive data. as a result, these 
data are printed out(log.info)
  
  

 


> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -------------------------------------------------------------
>
>                 Key: FLINK-20972
>                 URL: https://issues.apache.org/jira/browse/FLINK-20972
>             Project: Flink
>          Issue Type: Improvement
>          Components: API / DataStream
>    Affects Versions: 1.12.0
>         Environment: flink 1.4.0 +
>            Reporter: huajiewang
>            Priority: Minor
>              Labels: easyfix, pull-request-available
>         Attachments: Jdbc2PCSinkFunction.scala
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
> Maybe A large number of EventData will be output (log.info)
> ,which will cause IO bottleneck and disk waste
>  
>  my code in the attachment, A large number event data output in the log 
> output by flink, 
>   
>  like: 
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction 
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b, 
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ), 
> transactionStartTime=1610426158532} from checkpoint 4{code}
>  method notifyCheckpointComplete of TwoPhaseCommitSinkFunction,  about 
> LOG.info code is as follows:
> {code:java}
>   LOG.info("{} - checkpoint {} complete, committing transaction {} from 
> checkpoint {}", name(), checkpointId, pendingTransaction, 
> pendingTransactionCheckpointId);  
>  {code}
> will be call the toString method of pendingTransaction (TransactionHolder), 
> TransactionHolder'toString method code is:
>  
>  @Override
>  public String toString() {
>  return "TransactionHolder
> {" + "handle=" + handle + ", transactionStartTime=" + transactionStartTime + 
> '}
> ';
>  }
>   
>  handle is the concrete realization of my Transaction! There is a parameter 
> of List type in my Transaction, which is used to receive data. as a result, 
> these data are printed out(log.info)
>   
>   
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to