[jira] [Commented] (FLINK-21005) Introduce new provider for unified Sink API and implement in planner
[
https://issues.apache.org/jira/browse/FLINK-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17267062#comment-17267062
]
huajiewang commented on FLINK-21005:
please assign to me
> Introduce new provider for unified Sink API and implement in planner
>
>
> Key: FLINK-21005
> URL: https://issues.apache.org/jira/browse/FLINK-21005
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API, Table SQL / Planner
>Reporter: Jark Wu
>Priority: Major
> Fix For: 1.13.0
>
>
> FLIP-143 [1] introduced the unified sink API, we should add a
> {{SinkRuntimeProvider}} for it and support it in planner. So that Table SQL
> users can also use the unified sink APIs.
> [1]:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265817#comment-17265817
]
huajiewang edited comment on FLINK-20972 at 1/15/21, 8:39 AM:
--
[~gaoyunhaii]
My code is just an example,What you said makes sense, but in my opinion, this
is a notification message output by the flink framework to tell the user which
batch of checkpoint completed, but currently there are no requirements and
restrictions for the Transaction type, so that users can Free definition, a
little carelessness will cause this problem, unless the user is very familiar
with the processing logic of this code, in order to effectively avoid this
problem, about this Transaction class, if the flink output information when the
checkpoint is completed requires user participation, then Flink can completely
define an interface type (Transaction), allowing users to implement this
interface. So I think this is the issue of Flink
was (Author: benjobs):
[~gaoyunhaii] What you said is reasonable, but in my opinion, this is a
notification message output from the internal Flink framework, which is used to
tell the user which batch of checkpoint completed. However, at present, there
are no requirements and restrictions for the transaction type, so that the user
can freely define it. A little carelessness will cause this problem, unless the
user is very familiar with the processing logic of this code, In order to
effectively avoid this problem, regarding the transaction class, if the user is
required to participate in the output information when the checkpoint is
completed, the Flink can completely define an interface type (Transaction) for
the user to implement the interface. Therefore, I think this is the issue of
Flink
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> !1610682498960.jpg|width=838,height=630!
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance)
> TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265817#comment-17265817
]
huajiewang commented on FLINK-20972:
[~gaoyunhaii] What you said is reasonable, but in my opinion, this is a
notification message output from the internal Flink framework, which is used to
tell the user which batch of checkpoint completed. However, at present, there
are no requirements and restrictions for the transaction type, so that the user
can freely define it. A little carelessness will cause this problem, unless the
user is very familiar with the processing logic of this code, In order to
effectively avoid this problem, regarding the transaction class, if the user is
required to participate in the output information when the checkpoint is
completed, the Flink can completely define an interface type (Transaction) for
the user to implement the interface. Therefore, I think this is the issue of
Flink
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> !1610682498960.jpg|width=838,height=630!
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance)
> TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265732#comment-17265732
]
huajiewang edited comment on FLINK-20972 at 1/15/21, 6:29 AM:
--
Hi [~gaoyunhaii] u are right, my Transaction class like this:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString,
sqlList: ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sqlList += text
}{code}
java:
{code:java}
class Transaction {
private String transactionId;
private List sqlList;
public Transaction() {
this.transactionId = UUID.randomUUID().toString;
this.sqlList = new ArrayList<>;
}
public void add(String sql) {
this.sqlList.add(sql)
}
}{code}
log.info will be invoke Transaction'toString, all data of sqlList will be
output.
{code:java}
LOG.info( "{} - checkpoint {} complete, committing transaction {} from
checkpoint {}", name(), checkpointId, pendingTransaction,
pendingTransactionCheckpointId); {code}
was (Author: benjobs):
Hi [~gaoyunhaii] u are right, my Transaction class like this:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
{code:java}
class Transaction {
private String transactionId;
private List sqlList;
public Transaction() {
this.transactionId = UUID.randomUUID().toString;
this.sqlList = new ArrayList<>;
}
public void add(String sql) {
this.sqlList.add(sql)
}
}{code}
log.info will be invoke Transaction'toString, all data of sqlList will be
output.
{code:java}
LOG.info( "{} - checkpoint {} complete, committing transaction {} from
checkpoint {}", name(), checkpointId, pendingTransaction,
pendingTransactionCheckpointId); {code}
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> !1610682498960.jpg|width=838,height=630!
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance)
> TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265737#comment-17265737
]
huajiewang edited comment on FLINK-20972 at 1/15/21, 6:27 AM:
--
Hi [~gaoyunhaii] i can overwrite toString method in my Transaction, If don't
fix the TwoPhaseCommitSinkFunctionOutput, it means that all transaction class
may have such a problem. To solve this problem, you need to overwrite the
toString method in your Transaction class
was (Author: benjobs):
Hi [~gaoyunhaii] i can overwrite toString method in my Transaction, If don't
fix the TwoPhaseCommitSinkFunctionOutput, it means that all transaction class
may have such a problem. To solve this problem, you need to reset the toString
method.
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> !1610682498960.jpg|width=838,height=630!
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance)
> TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265737#comment-17265737
]
huajiewang edited comment on FLINK-20972 at 1/15/21, 6:25 AM:
--
Hi [~gaoyunhaii] i can overwrite toString method in my Transaction, If don't
fix the TwoPhaseCommitSinkFunctionOutput, it means that all transaction class
may have such a problem. To solve this problem, you need to reset the toString
method.
was (Author: benjobs):
Hi [~gaoyunhaii] i can overwrite toString method for my Transaction, If don't
fix the TwoPhaseCommitSinkFunctionOutput, it means that all transaction class
may have such a problem. To solve this problem, you need to reset the toString
method.
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> !1610682498960.jpg|width=838,height=630!
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance)
> TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265737#comment-17265737
]
huajiewang commented on FLINK-20972:
Hi [~gaoyunhaii] i can overwrite toString method for my Transaction, If don't
fix the TwoPhaseCommitSinkFunctionOutput, it means that all transaction class
may have such a problem. To solve this problem, you need to reset the toString
method.
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> !1610682498960.jpg|width=838,height=630!
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance)
> TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265732#comment-17265732
]
huajiewang edited comment on FLINK-20972 at 1/15/21, 6:20 AM:
--
Hi [~gaoyunhaii] u are right, my Transaction class like this:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
{code:java}
class Transaction {
private String transactionId;
private List sqlList;
public Transaction() {
this.transactionId = UUID.randomUUID().toString;
this.sqlList = new ArrayList<>;
}
public void add(String sql) {
this.sqlList.add(sql)
}
}{code}
log.info will be invoke Transaction'toString, all data of sqlList will be
output.
{code:java}
LOG.info( "{} - checkpoint {} complete, committing transaction {} from
checkpoint {}", name(), checkpointId, pendingTransaction,
pendingTransactionCheckpointId); {code}
was (Author: benjobs):
Hi [~gaoyunhaii] u are right, my Transaction class like this:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
{code:java}
class Transaction {
private String transactionId;
private List sqlList;
public Transaction() {
this.transactionId = UUID.randomUUID().toString;
this.sqlList = new ArrayList<>;
}
public void add(String sql) {
this.sqlList.add(sql)
}
}{code}
log.info will be invoke Transaction'toString
{code:java}
LOG.info( "{} - checkpoint {} complete, committing transaction {} from
checkpoint {}", name(), checkpointId, pendingTransaction,
pendingTransactionCheckpointId); {code}
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> !1610682498960.jpg|width=838,height=630!
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance)
> TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265732#comment-17265732
]
huajiewang edited comment on FLINK-20972 at 1/15/21, 6:19 AM:
--
Hi [~gaoyunhaii] u are right, my Transaction class like this:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
{code:java}
class Transaction {
private String transactionId;
private List sqlList;
public Transaction() {
this.transactionId = UUID.randomUUID().toString;
this.sqlList = new ArrayList<>;
}
public void add(String sql) {
this.sqlList.add(sql)
}
}{code}
log.info will be invoke Transaction'toString
{code:java}
LOG.info( "{} - checkpoint {} complete, committing transaction {} from
checkpoint {}", name(), checkpointId, pendingTransaction,
pendingTransactionCheckpointId); {code}
was (Author: benjobs):
Hi [~gaoyunhaii] u are right, my Transaction class like this:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
{code:java}
class Transaction {
private String transactionId;
private List sqlList;
public Transaction() {
this.transactionId = UUID.randomUUID().toString;
this.sqlList = new ArrayList<>;
}
public void add(String sql) {
this.sqlList.add(sql)
}
}{code}
log.info will be invoke Transaction'toString
{code:java}
LOG.info( "{} - checkpoint {} complete, committing transaction {} from
checkpoint {}", name(), checkpointId, pendingTransaction,
pendingTransactionCheckpointId); {code}
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> !1610682498960.jpg|width=838,height=630!
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance)
> TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265732#comment-17265732
]
huajiewang edited comment on FLINK-20972 at 1/15/21, 6:18 AM:
--
Hi [~gaoyunhaii] u are right, my Transaction class like this:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
{code:java}
class Transaction {
private String transactionId;
private List sqlList;
public Transaction() {
this.transactionId = UUID.randomUUID().toString;
this.sqlList = new ArrayList<>;
}
public void add(String sql) {
this.sqlList.add(sql)
}
}{code}
log.info will be invoke Transaction'toString
{code:java}
LOG.info( "{} - checkpoint {} complete, committing transaction {} from
checkpoint {}", name(), checkpointId, pendingTransaction,
pendingTransactionCheckpointId); {code}
was (Author: benjobs):
Hi [~gaoyunhaii] u are right, my Transaction class like this:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
{code:java}
class Transaction {
private String transactionId;
private List sqlList;
public Transaction() {
this.transactionId = UUID.randomUUID().toString;
this.sqlList = new ArrayList<>;
}
public void add(String sql) {
this.sqlList.add(sql)
}
}{code}
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> !1610682498960.jpg|width=838,height=630!
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance)
> TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265732#comment-17265732
]
huajiewang edited comment on FLINK-20972 at 1/15/21, 6:18 AM:
--
Hi [~gaoyunhaii] u are right, my Transaction class like this:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
{code:java}
class Transaction {
private String transactionId;
private List sqlList;
public Transaction() {
this.transactionId = UUID.randomUUID().toString;
this.sqlList = new ArrayList<>;
}
public void add(String sql) {
this.sqlList.add(sql)
}
}{code}
log.info will be invoke Transaction'toString
{code:java}
LOG.info( "{} - checkpoint {} complete, committing transaction {} from
checkpoint {}", name(), checkpointId, pendingTransaction,
pendingTransactionCheckpointId); {code}
was (Author: benjobs):
Hi [~gaoyunhaii] u are right, my Transaction class like this:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
{code:java}
class Transaction {
private String transactionId;
private List sqlList;
public Transaction() {
this.transactionId = UUID.randomUUID().toString;
this.sqlList = new ArrayList<>;
}
public void add(String sql) {
this.sqlList.add(sql)
}
}{code}
log.info will be invoke Transaction'toString
{code:java}
LOG.info( "{} - checkpoint {} complete, committing transaction {} from
checkpoint {}", name(), checkpointId, pendingTransaction,
pendingTransactionCheckpointId); {code}
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> !1610682498960.jpg|width=838,height=630!
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance)
> TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265732#comment-17265732
]
huajiewang edited comment on FLINK-20972 at 1/15/21, 6:18 AM:
--
Hi [~gaoyunhaii] u are right, my Transaction class like this:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
{code:java}
class Transaction {
private String transactionId;
private List sqlList;
public Transaction() {
this.transactionId = UUID.randomUUID().toString;
this.sqlList = new ArrayList<>;
}
public void add(String sql) {
this.sqlList.add(sql)
}
}{code}
log.info will be invoke Transaction'toString
{code:java}
LOG.info( "{} - checkpoint {} complete, committing transaction {} from
checkpoint {}", name(), checkpointId, pendingTransaction,
pendingTransactionCheckpointId); {code}
was (Author: benjobs):
Hi [~gaoyunhaii] u are right, my Transaction class like this:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
{code:java}
class Transaction {
private String transactionId;
private List sqlList;
public Transaction() {
this.transactionId = UUID.randomUUID().toString;
this.sqlList = new ArrayList<>;
}
public void add(String sql) {
this.sqlList.add(sql)
}
}{code}
log.info will be invoke Transaction'toString
{code:java}
LOG.info( "{} - checkpoint {} complete, committing transaction {} from
checkpoint {}", name(), checkpointId, pendingTransaction,
pendingTransactionCheckpointId); {code}
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> !1610682498960.jpg|width=838,height=630!
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance)
> TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265732#comment-17265732
]
huajiewang edited comment on FLINK-20972 at 1/15/21, 6:17 AM:
--
Hi [~gaoyunhaii] u are right, my Transaction class like this:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
{code:java}
class Transaction {
private String transactionId;
private List sqlList;
public Transaction() {
this.transactionId = UUID.randomUUID().toString;
this.sqlList = new ArrayList<>;
}
public void add(String sql) {
this.sqlList.add(sql)
}
}{code}
was (Author: benjobs):
Hi [~gaoyunhaii] u are right, my Transaction class like this:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
{code:java}
class Transaction {
private String transactionId;
private List sqlList;
public Transaction() {
this.transactionId = UUID.randomUUID().toString;
this.sqlList = new ArrayList<>;
}
public void add(String sql) {
this.sqlList.add(sql)
}
}{code}
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> !1610682498960.jpg|width=838,height=630!
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance)
> TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265732#comment-17265732
]
huajiewang edited comment on FLINK-20972 at 1/15/21, 6:16 AM:
--
Hi [~gaoyunhaii] u are right, my Transaction class like this:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
{code:java}
class Transaction {
private String transactionId;
private List sqlList;
public Transaction() {
this.transactionId = UUID.randomUUID().toString;
this.sqlList = new ArrayList<>;
}
public void add(String sql) {
this.sqlList.add(sql)
}
}{code}
was (Author: benjobs):
u are right, my Transaction class is here:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> !1610682498960.jpg|width=838,height=630!
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance)
> TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265732#comment-17265732
]
huajiewang commented on FLINK-20972:
u are right, my Transaction class is here:
scala:
{code:java}
case class Transaction(transactionId: String = UUID.randomUUID().toString, sql:
ListBuffer[String] = ListBuffer.empty) extends Serializable {
def +(text: String): Unit = sql += text
}{code}
java:
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> !1610682498960.jpg|width=838,height=630!
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance)
> TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Description:
in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
output(log.info),which will cause IO bottleneck and disk waste.
my code in the attachment, A large number event data output in the log output
by flink , e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
!1610682498960.jpg|width=838,height=630!
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
will be invoke pendingTransaction'toString method (pendingTransaction is
TransactionHolder'instance)
TransactionHolder'toString method code is:
!1610682603148.jpg|width=859,height=327!
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
was:
in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
output(log.info),which will cause IO bottleneck and disk waste.
my code in the attachment, A large number event data output in the log output
by flink , e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
!1610682498960.jpg|width=838,height=630!
will be invoke pendingTransaction'toString method (pendingTransaction is
TransactionHolder'instance), TransactionHolder'toString method code is:
!1610682603148.jpg|width=859,height=327!
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> !1610682498960.jpg|width=838,height=630!
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance)
> TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Description:
in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
output(log.info),which will cause IO bottleneck and disk waste.
my code in the attachment, A large number event data output in the log output
by flink , e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
!1610682498960.jpg|width=838,height=630!
will be invoke pendingTransaction'toString method (pendingTransaction is
TransactionHolder'instance), TransactionHolder'toString method code is:
!1610682603148.jpg|width=859,height=327!
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
was:
in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
output(log.info),which will cause IO bottleneck and disk waste.
my code in the attachment, A large number event data output in the log output
by flink , e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
will be invoke pendingTransaction'toString method (pendingTransaction is
TransactionHolder'instance), TransactionHolder'toString method code is:
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> !1610682498960.jpg|width=838,height=630!
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance), TransactionHolder'toString method code is:
> !1610682603148.jpg|width=859,height=327!
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> +
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Attachment: 1610682603148.jpg
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, 1610682603148.jpg,
> Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance), TransactionHolder'toString method code is:
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Attachment: 1610682498960.jpg
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: 1610682498960.jpg, Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance), TransactionHolder'toString method code is:
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Description:
in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
output(log.info),which will cause IO bottleneck and disk waste.
my code in the attachment, A large number event data output in the log output
by flink , e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
will be invoke pendingTransaction'toString method (pendingTransaction is
TransactionHolder'instance), TransactionHolder'toString method code is:
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
was:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, Maybe A
large number of EventData will be output (log.info),which will cause IO
bottleneck and disk waste.
my code in the attachment, A large number event data output in the log output
by flink , e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
will be invoke pendingTransaction'toString method (pendingTransaction is
TransactionHolder'instance), TransactionHolder'toString method code is:
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be
> output(log.info),which will cause IO bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance), TransactionHolder'toString method code is:
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
> handle is the concrete realization of my Transaction!
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Description:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, Maybe A
large number of EventData will be output (log.info),which will cause IO
bottleneck and disk waste.
my code in the attachment, A large number event data output in the log output
by flink , e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
will be invoke pendingTransaction'toString method (pendingTransaction is
TransactionHolder'instance), TransactionHolder'toString method code is:
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
was:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, Maybe A
large number of EventData will be output (log.info),which will cause IO
bottleneck and disk waste.
my code in the attachment, A large number event data output in the log output
by flink , e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
will be invoke pendingTransaction'toString method (pendingTransaction is
TransactionHolder'instance), TransactionHolder'toString method code is:
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, Maybe
> A large number of EventData will be output (log.info),which will cause IO
> bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance), TransactionHolder'toString method code is:
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
>
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Description:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, Maybe A
large number of EventData will be output (log.info),which will cause IO
bottleneck and disk waste.
my code in the attachment, A large number event data output in the log output
by flink , e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
will be invoke pendingTransaction'toString method (pendingTransaction is
TransactionHolder'instance), TransactionHolder'toString method code is:
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
was:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, Maybe A
large number of EventData will be output (log.info),which will cause IO
bottleneck and disk waste.
my code in the attachment, A large number event data output in the log output
by flink , e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
will be invoke pendingTransaction'toString method (pendingTransaction is
TransactionHolder'instance), TransactionHolder'toString method code is:
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, Maybe
> A large number of EventData will be output (log.info),which will cause IO
> bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
>
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
>
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance), TransactionHolder'toString method code is:
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> +
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Description:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, Maybe A
large number of EventData will be output (log.info),which will cause IO
bottleneck and disk waste.
my code in the attachment, A large number event data output in the log output
by flink , e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
will be invoke pendingTransaction'toString method (pendingTransaction is
TransactionHolder'instance), TransactionHolder'toString method code is:
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
was:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, Maybe A
large number of EventData will be output (log.info),which will cause IO
bottleneck and disk waste.
my code in the attachment, A large number event data output in the log output
by flink , output e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
will be invoke pendingTransaction'toString method (pendingTransaction is
TransactionHolder'instance), TransactionHolder'toString method code is:
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, Maybe
> A large number of EventData will be output (log.info),which will cause IO
> bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance), TransactionHolder'toString method code is:
>
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> +
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Description:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, Maybe A
large number of EventData will be output (log.info),which will cause IO
bottleneck and disk waste.
my code in the attachment, A large number event data output in the log output
by flink , output e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
will be invoke pendingTransaction'toString method (pendingTransaction is
TransactionHolder'instance), TransactionHolder'toString method code is:
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
was:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
Maybe A large number of EventData will be output (log.info)
,which will cause IO bottleneck and disk waste
my code in the attachment, A large number event data output in the log output
by flink,
e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
will be invoke pendingTransaction'toString method (pendingTransaction is
TransactionHolder'instance), TransactionHolder'toString method code is:
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, Maybe
> A large number of EventData will be output (log.info),which will cause IO
> bottleneck and disk waste.
>
> my code in the attachment, A large number event data output in the log
> output by flink , output e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance), TransactionHolder'toString method code is:
>
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> +
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Description:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
Maybe A large number of EventData will be output (log.info)
,which will cause IO bottleneck and disk waste
my code in the attachment, A large number event data output in the log output
by flink,
e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
will be invoke pendingTransaction'toString method (pendingTransaction is
TransactionHolder'instance), TransactionHolder'toString method code is:
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
was:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
Maybe A large number of EventData will be output (log.info)
,which will cause IO bottleneck and disk waste
my code in the attachment, A large number event data output in the log output
by flink,
e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
method notifyCheckpointComplete of TwoPhaseCommitSinkFunction, about LOG.info
code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
will be call the toString method of pendingTransaction (TransactionHolder),
TransactionHolder'toString method code is:
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
> Maybe A large number of EventData will be output (log.info)
> ,which will cause IO bottleneck and disk waste
>
> my code in the attachment, A large number event data output in the log
> output by flink,
>
> e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be invoke pendingTransaction'toString method (pendingTransaction is
> TransactionHolder'instance), TransactionHolder'toString method code is:
>
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
>
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Description:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
Maybe A large number of EventData will be output (log.info)
,which will cause IO bottleneck and disk waste
my code in the attachment, A large number event data output in the log output
by flink,
e.g:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
method notifyCheckpointComplete of TwoPhaseCommitSinkFunction, about LOG.info
code is as follows:
{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint
{}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId); {code}
will be call the toString method of pendingTransaction (TransactionHolder),
TransactionHolder'toString method code is:
{code:java}
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}{code}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
was:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
Maybe A large number of EventData will be output (log.info)
,which will cause IO bottleneck and disk waste
my code in the attachment, A large number event data output in the log output
by flink,
like:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
method notifyCheckpointComplete of TwoPhaseCommitSinkFunction, about LOG.info
code is as follows:
{code:java}
LOG.info("{} - checkpoint {} complete, committing transaction {} from
checkpoint {}", name(), checkpointId, pendingTransaction,
pendingTransactionCheckpointId);
{code}
will be call the toString method of pendingTransaction (TransactionHolder),
TransactionHolder'toString method code is:
@Override
public String toString() {
return "TransactionHolder
{" + "handle=" + handle + ", transactionStartTime=" + transactionStartTime + '}
';
}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
> Maybe A large number of EventData will be output (log.info)
> ,which will cause IO bottleneck and disk waste
>
> my code in the attachment, A large number event data output in the log
> output by flink,
>
> e.g:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> method notifyCheckpointComplete of TwoPhaseCommitSinkFunction, about
> LOG.info code is as follows:
> {code:java}
> LOG.info(
> "{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(),
> checkpointId,
> pendingTransaction,
> pendingTransactionCheckpointId); {code}
> will be call the toString method of pendingTransaction (TransactionHolder),
> TransactionHolder'toString method code is:
>
> {code:java}
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }{code}
>
> handle is the
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Description:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
Maybe A large number of EventData will be output (log.info)
,which will cause IO bottleneck and disk waste
my code in the attachment, A large number event data output in the log output
by flink,
like:
{code:java}
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4{code}
method notifyCheckpointComplete of TwoPhaseCommitSinkFunction, about LOG.info
code is as follows:
{code:java}
LOG.info("{} - checkpoint {} complete, committing transaction {} from
checkpoint {}", name(), checkpointId, pendingTransaction,
pendingTransactionCheckpointId);
{code}
will be call the toString method of pendingTransaction (TransactionHolder),
TransactionHolder'toString method code is:
@Override
public String toString() {
return "TransactionHolder
{" + "handle=" + handle + ", transactionStartTime=" + transactionStartTime + '}
';
}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
was:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
Maybe A large number of EventData will be output (log.info)
,which will cause IO bottleneck and disk waste
my code in the attachment, A large number event data output in the log output
by flink,
like:
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde \{handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4
method notifyCheckpointComplete of TwoPhaseCommitSinkFunction, about LOG.info
code is as follows:
LOG.info("{} - checkpoint {} complete, committing transaction {} from
checkpoint {}",
name(), checkpointId, pendingTransaction, pendingTransactionCheckpointId);
will be call the toString method of pendingTransaction (TransactionHolder),
TransactionHolder'toString method code is:
@Override
public String toString() {
return "TransactionHolder
{" + "handle=" + handle + ", transactionStartTime=" + transactionStartTime + '}
';
}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
> Maybe A large number of EventData will be output (log.info)
> ,which will cause IO bottleneck and disk waste
>
> my code in the attachment, A large number event data output in the log
> output by flink,
>
> like:
> {code:java}
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4{code}
> method notifyCheckpointComplete of TwoPhaseCommitSinkFunction, about
> LOG.info code is as follows:
> {code:java}
> LOG.info("{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}", name(), checkpointId, pendingTransaction,
> pendingTransactionCheckpointId);
> {code}
> will be call the toString method of pendingTransaction (TransactionHolder),
> TransactionHolder'toString method code is:
>
> @Override
> public String toString() {
> return "TransactionHolder
> {" + "handle=" + handle + ", transactionStartTime=" + transactionStartTime +
> '}
> ';
> }
>
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Description:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
Maybe A large number of EventData will be output (log.info)
,which will cause IO bottleneck and disk waste
my code in the attachment, A large number event data output in the log output
by flink,
like:
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde \{handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4
method notifyCheckpointComplete of TwoPhaseCommitSinkFunction, about LOG.info
code is as follows:
LOG.info("{} - checkpoint {} complete, committing transaction {} from
checkpoint {}",
name(), checkpointId, pendingTransaction, pendingTransactionCheckpointId);
will be call the toString method of pendingTransaction (TransactionHolder),
TransactionHolder'toString method code is:
@Override
public String toString() {
return "TransactionHolder
{" + "handle=" + handle + ", transactionStartTime=" + transactionStartTime + '}
';
}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
was:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
Maybe A large number of EventData will be output (log.info)
,which will cause IO bottleneck and disk waste
my code in the attachment, A large number event data output in the log output
by flink,
like:
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde
{handle=Transaction(b420c880a951403984f231dd7e33597b, ListBuffer(insert into
table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532}
from checkpoint 4
method notifyCheckpointComplete of TwoPhaseCommitSinkFunction, about LOG.info
code is as follows:
LOG.info("{} - checkpoint {} complete, committing transaction {} from
checkpoint {}",
name(), checkpointId, pendingTransaction, pendingTransactionCheckpointId);
will be call the toString method of pendingTransaction (TransactionHolder),
TransactionHolder'toString method code is:
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
> Maybe A large number of EventData will be output (log.info)
> ,which will cause IO bottleneck and disk waste
>
> my code in the attachment, A large number event data output in the log
> output by flink,
>
> like:
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde \{handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4
>
> method notifyCheckpointComplete of TwoPhaseCommitSinkFunction, about
> LOG.info code is as follows:
> LOG.info("{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(), checkpointId, pendingTransaction, pendingTransactionCheckpointId);
>
> will be call the toString method of pendingTransaction (TransactionHolder),
> TransactionHolder'toString method code is:
> @Override
> public String toString() {
> return "TransactionHolder
> {" + "handle=" + handle + ", transactionStartTime=" + transactionStartTime +
> '}
> ';
> }
>
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264793#comment-17264793
]
huajiewang commented on FLINK-20972:
yes!
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix, pull-request-available
> Attachments: Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
> Maybe A large number of EventData will be output (log.info)
> ,which will cause IO bottleneck and disk waste
>
> my code in the attachment, A large number event data output in the log
> output by flink,
>
> like:
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde \{handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4
>
> method notifyCheckpointComplete of TwoPhaseCommitSinkFunction, about
> LOG.info code is as follows:
> LOG.info("{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(), checkpointId, pendingTransaction, pendingTransactionCheckpointId);
>
> will be call the toString method of pendingTransaction (TransactionHolder),
> TransactionHolder'toString method code is:
> @Override
> public String toString() {
> return "TransactionHolder
> {" + "handle=" + handle + ", transactionStartTime=" + transactionStartTime +
> '}
> ';
> }
>
> handle is the concrete realization of my Transaction! There is a parameter
> of List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Description:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
Maybe A large number of EventData will be output (log.info)
,which will cause IO bottleneck and disk waste
my code in the attachment, A large number event data output in the log output
by flink,
like:
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolde
{handle=Transaction(b420c880a951403984f231dd7e33597b, ListBuffer(insert into
table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532}
from checkpoint 4
method notifyCheckpointComplete of TwoPhaseCommitSinkFunction, about LOG.info
code is as follows:
LOG.info("{} - checkpoint {} complete, committing transaction {} from
checkpoint {}",
name(), checkpointId, pendingTransaction, pendingTransactionCheckpointId);
will be call the toString method of pendingTransaction (TransactionHolder),
TransactionHolder'toString method code is:
@Override
public String toString() {
return "TransactionHolder{"
+ "handle="
+ handle
+ ", transactionStartTime="
+ transactionStartTime
+ '}';
}
handle is the concrete realization of my Transaction! There is a parameter of
List type in my Transaction, which is used to receive data. as a result, these
data are printed out(log.info)
was:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
Maybe A large number of EventData will be output (log.info)
,which will cause IO bottleneck and disk waste
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolder{handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix
> Attachments: Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
> Maybe A large number of EventData will be output (log.info)
> ,which will cause IO bottleneck and disk waste
>
> my code in the attachment, A large number event data output in the log output
> by flink,
>
> like:
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolde
> {handle=Transaction(b420c880a951403984f231dd7e33597b, ListBuffer(insert into
> table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532}
> from checkpoint 4
>
>
> method notifyCheckpointComplete of TwoPhaseCommitSinkFunction, about
> LOG.info code is as follows:
> LOG.info("{} - checkpoint {} complete, committing transaction {} from
> checkpoint {}",
> name(), checkpointId, pendingTransaction, pendingTransactionCheckpointId);
>
> will be call the toString method of pendingTransaction (TransactionHolder),
> TransactionHolder'toString method code is:
> @Override
> public String toString() {
> return "TransactionHolder{"
> + "handle="
> + handle
> + ", transactionStartTime="
> + transactionStartTime
> + '}';
> }
>
> handle is the concrete realization of my Transaction! There is a parameter of
> List type in my Transaction, which is used to receive data. as a result,
> these data are printed out(log.info)
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[
https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huajiewang updated FLINK-20972:
---
Description:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
Maybe A large number of EventData will be output (log.info)
,which will cause IO bottleneck and disk waste
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
TransactionHolder{handle=Transaction(b420c880a951403984f231dd7e33597b,
ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
transactionStartTime=1610426158532} from checkpoint 4
was:
when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
Maybe A large number of EventData will be output (log.info)
,which will cause IO bottleneck and disk waste
> TwoPhaseCommitSinkFunction Output a large amount of EventData
> -
>
> Key: FLINK-20972
> URL: https://issues.apache.org/jira/browse/FLINK-20972
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
> Environment: flink 1.4.0 +
>Reporter: huajiewang
>Priority: Minor
> Labels: easyfix
> Attachments: Jdbc2PCSinkFunction.scala
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete,
> Maybe A large number of EventData will be output (log.info)
> ,which will cause IO bottleneck and disk waste
>
>
> Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction
> TransactionHolder{handle=Transaction(b420c880a951403984f231dd7e33597b,
> ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ),
> transactionStartTime=1610426158532} from checkpoint 4
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
[ https://issues.apache.org/jira/browse/FLINK-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huajiewang updated FLINK-20972: --- Attachment: Jdbc2PCSinkFunction.scala > TwoPhaseCommitSinkFunction Output a large amount of EventData > - > > Key: FLINK-20972 > URL: https://issues.apache.org/jira/browse/FLINK-20972 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.12.0 > Environment: flink 1.4.0 + >Reporter: huajiewang >Priority: Minor > Labels: easyfix > Attachments: Jdbc2PCSinkFunction.scala > > Original Estimate: 1h > Remaining Estimate: 1h > > when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, > Maybe A large number of EventData will be output (log.info) > ,which will cause IO bottleneck and disk waste > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20972) TwoPhaseCommitSinkFunction Output a large amount of EventData
huajiewang created FLINK-20972: -- Summary: TwoPhaseCommitSinkFunction Output a large amount of EventData Key: FLINK-20972 URL: https://issues.apache.org/jira/browse/FLINK-20972 Project: Flink Issue Type: Improvement Components: API / DataStream Affects Versions: 1.12.0 Environment: flink 1.4.0 + Reporter: huajiewang when TwoPhaseCommitSinkFunctionOutput tigger notifyCheckpointComplete, Maybe A large number of EventData will be output (log.info) ,which will cause IO bottleneck and disk waste -- This message was sent by Atlassian Jira (v8.3.4#803005)
