[jira] [Commented] (FLINK-24314) Always use memory state backend with RocksDB
[
https://issues.apache.org/jira/browse/FLINK-24314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416598#comment-17416598
]
shiwuliang commented on FLINK-24314:
However, the content in the red box actually tells flink that I want to use fs
to store the checkpoint, but it does not actually take effect. While not taking
effect, it also affected the statebackend and checkpoint storage I set for
streamEnv.
> Always use memory state backend with RocksDB
>
>
> Key: FLINK-24314
> URL: https://issues.apache.org/jira/browse/FLINK-24314
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: shiwuliang
>Priority: Major
> Attachments: image-2021-09-17-10-59-50-094.png,
> image-2021-09-17-12-03-58-285.png, image-2021-09-17-12-04-55-009.png,
> image-2021-09-17-12-16-46-277.png, image-2021-09-17-17-46-07-196.png,
> image-2021-09-17-17-47-20-950.png, image-2021-09-17-17-50-19-099.png
>
>
> When I config to use `RocksDBStatebackend`,
>
> {code:java}
> //代码占位符
> RocksDBStateBackend rocksDBStateBackend = new
> RocksDBStateBackend("hdfs://");
> streamEnv.setStateBackend(rocksDBStateBackend);{code}
>
> there are some exception like this:
> !image-2021-09-17-10-59-50-094.png|width=1446,height=420!
>
> Seems like the RocksdbStatebackend will use FsStateBackend to store
> checkpoints. So it means that I have used FileSystemStateBackend, why does
> this exception occur?
>
> I use flink 1.13.0 and found a similar question like this at:
> [https://stackoverflow.com/questions/68314652/flink-state-backend-config-with-the-state-processor-api]
>
> I'm not sure if his question is same with mine.
> I want to know how can I solve this and if it is indeed a 1.13.0 bug, how can
> I bypass it besides upgrading?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24314) Always use memory state backend with RocksDB
[
https://issues.apache.org/jira/browse/FLINK-24314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416595#comment-17416595
]
shiwuliang commented on FLINK-24314:
I think this is a bug. When I tested locally, I found a phenomenon, when I
removed the content of the red box in the picture, fs will be used to store the
checkpoint. When I add it, memory will be used to store the checkpoint.
!image-2021-09-17-17-46-07-196.png!
> Always use memory state backend with RocksDB
>
>
> Key: FLINK-24314
> URL: https://issues.apache.org/jira/browse/FLINK-24314
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: shiwuliang
>Priority: Major
> Attachments: image-2021-09-17-10-59-50-094.png,
> image-2021-09-17-12-03-58-285.png, image-2021-09-17-12-04-55-009.png,
> image-2021-09-17-12-16-46-277.png, image-2021-09-17-17-46-07-196.png,
> image-2021-09-17-17-47-20-950.png
>
>
> When I config to use `RocksDBStatebackend`,
>
> {code:java}
> //代码占位符
> RocksDBStateBackend rocksDBStateBackend = new
> RocksDBStateBackend("hdfs://");
> streamEnv.setStateBackend(rocksDBStateBackend);{code}
>
> there are some exception like this:
> !image-2021-09-17-10-59-50-094.png|width=1446,height=420!
>
> Seems like the RocksdbStatebackend will use FsStateBackend to store
> checkpoints. So it means that I have used FileSystemStateBackend, why does
> this exception occur?
>
> I use flink 1.13.0 and found a similar question like this at:
> [https://stackoverflow.com/questions/68314652/flink-state-backend-config-with-the-state-processor-api]
>
> I'm not sure if his question is same with mine.
> I want to know how can I solve this and if it is indeed a 1.13.0 bug, how can
> I bypass it besides upgrading?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24314) Always use memory state backend with RocksDB
[
https://issues.apache.org/jira/browse/FLINK-24314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416572#comment-17416572
]
zlzhang0122 commented on FLINK-24314:
-
[~shiwuliang] You can set like this:
{code:java}
env.setStateBackend(new EmbeddedRocksDBStateBackend());
env.getCheckpointConfig().setCheckpointStorage(checkpointDir);
{code}
Using embedded rocksdb to store state, and using checkpointDir to store
checkpoint.
> Always use memory state backend with RocksDB
>
>
> Key: FLINK-24314
> URL: https://issues.apache.org/jira/browse/FLINK-24314
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: shiwuliang
>Priority: Major
> Attachments: image-2021-09-17-10-59-50-094.png,
> image-2021-09-17-12-03-58-285.png, image-2021-09-17-12-04-55-009.png,
> image-2021-09-17-12-16-46-277.png
>
>
> When I config to use `RocksDBStatebackend`,
>
> {code:java}
> //代码占位符
> RocksDBStateBackend rocksDBStateBackend = new
> RocksDBStateBackend("hdfs://");
> streamEnv.setStateBackend(rocksDBStateBackend);{code}
>
> there are some exception like this:
> !image-2021-09-17-10-59-50-094.png|width=1446,height=420!
>
> Seems like the RocksdbStatebackend will use FsStateBackend to store
> checkpoints. So it means that I have used FileSystemStateBackend, why does
> this exception occur?
>
> I use flink 1.13.0 and found a similar question like this at:
> [https://stackoverflow.com/questions/68314652/flink-state-backend-config-with-the-state-processor-api]
>
> I'm not sure if his question is same with mine.
> I want to know how can I solve this and if it is indeed a 1.13.0 bug, how can
> I bypass it besides upgrading?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24314) Always use memory state backend with RocksDB
[
https://issues.apache.org/jira/browse/FLINK-24314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416430#comment-17416430
]
shiwuliang commented on FLINK-24314:
[~zlzhang0122]
by the way, do u mean that I should config like this?
!image-2021-09-17-12-16-46-277.png!
> Always use memory state backend with RocksDB
>
>
> Key: FLINK-24314
> URL: https://issues.apache.org/jira/browse/FLINK-24314
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: shiwuliang
>Priority: Major
> Attachments: image-2021-09-17-10-59-50-094.png,
> image-2021-09-17-12-03-58-285.png, image-2021-09-17-12-04-55-009.png,
> image-2021-09-17-12-16-46-277.png
>
>
> When I config to use `RocksDBStatebackend`,
>
> {code:java}
> //代码占位符
> RocksDBStateBackend rocksDBStateBackend = new
> RocksDBStateBackend("hdfs://");
> streamEnv.setStateBackend(rocksDBStateBackend);{code}
>
> there are some exception like this:
> !image-2021-09-17-10-59-50-094.png|width=1446,height=420!
>
> Seems like the RocksdbStatebackend will use FsStateBackend to store
> checkpoints. So it means that I have used FileSystemStateBackend, why does
> this exception occur?
>
> I use flink 1.13.0 and found a similar question like this at:
> [https://stackoverflow.com/questions/68314652/flink-state-backend-config-with-the-state-processor-api]
>
> I'm not sure if his question is same with mine.
> I want to know how can I solve this and if it is indeed a 1.13.0 bug, how can
> I bypass it besides upgrading?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24314) Always use memory state backend with RocksDB
[
https://issues.apache.org/jira/browse/FLINK-24314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416427#comment-17416427
]
shiwuliang commented on FLINK-24314:
[~zlzhang0122] hi, thanks for your reply.
But in RocksDBStateBackend's constructor, it create a FsStateBackend to store
checkpoint data:
!image-2021-09-17-12-04-55-009.png!
I use `new RocksDBStateBackend("hdfs://")` to create a RocksDBStateBackend,
seems like the 'hdfs://' is the checkpoint storage location...
> Always use memory state backend with RocksDB
>
>
> Key: FLINK-24314
> URL: https://issues.apache.org/jira/browse/FLINK-24314
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: shiwuliang
>Priority: Major
> Attachments: image-2021-09-17-10-59-50-094.png,
> image-2021-09-17-12-03-58-285.png, image-2021-09-17-12-04-55-009.png
>
>
> When I config to use `RocksDBStatebackend`,
>
> {code:java}
> //代码占位符
> RocksDBStateBackend rocksDBStateBackend = new
> RocksDBStateBackend("hdfs://");
> streamEnv.setStateBackend(rocksDBStateBackend);{code}
>
> there are some exception like this:
> !image-2021-09-17-10-59-50-094.png|width=1446,height=420!
>
> Seems like the RocksdbStatebackend will use FsStateBackend to store
> checkpoints. So it means that I have used FileSystemStateBackend, why does
> this exception occur?
>
> I use flink 1.13.0 and found a similar question like this at:
> [https://stackoverflow.com/questions/68314652/flink-state-backend-config-with-the-state-processor-api]
>
> I'm not sure if his question is same with mine.
> I want to know how can I solve this and if it is indeed a 1.13.0 bug, how can
> I bypass it besides upgrading?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24314) Always use memory state backend with RocksDB
[
https://issues.apache.org/jira/browse/FLINK-24314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416426#comment-17416426
]
zlzhang0122 commented on FLINK-24314:
-
In Flink 1.13, the storage of the state and checkpoint was separated, so you
should set statebackend and checkpoint location at the same time. In your case,
you just set the statebackend, so it use memory to store the checkpoint as
default and since the size is beyound the limit, it report an exception.
> Always use memory state backend with RocksDB
>
>
> Key: FLINK-24314
> URL: https://issues.apache.org/jira/browse/FLINK-24314
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: shiwuliang
>Priority: Major
> Attachments: image-2021-09-17-10-59-50-094.png
>
>
> When I config to use `RocksDBStatebackend`,
>
> {code:java}
> //代码占位符
> RocksDBStateBackend rocksDBStateBackend = new
> RocksDBStateBackend("hdfs://");
> streamEnv.setStateBackend(rocksDBStateBackend);{code}
>
> there are some exception like this:
> !image-2021-09-17-10-59-50-094.png|width=1446,height=420!
>
> Seems like the RocksdbStatebackend will use FsStateBackend to store
> checkpoints. So it means that I have used FileSystemStateBackend, why does
> this exception occur?
>
> I use flink 1.13.0 and found a similar question like this at:
> [https://stackoverflow.com/questions/68314652/flink-state-backend-config-with-the-state-processor-api]
>
> I'm not sure if his question is same with mine.
> I want to know how can I solve this and if it is indeed a 1.13.0 bug, how can
> I bypass it besides upgrading?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
