建议通过 HADOOP_HOME 或 HADOOP_CONF_DIR 环境配置，Flink 有一个 fallback 的加载优先级
1. HADOOP_HOME
2. Configuration 也就是 fs.hdfs.hadoopconf
3. HADOOP_CONF_DIR
其中 Configuration 的方式是已废弃的
Best,
tison.
LJY 于2020年1月9日周四 下午3:52写道：
> 各位好：
>
> 目前hadoop的配置文件是在 fs.hdfs.hadoopconf 设置。
>
> 用户是否能够不启用配置文件中的fs.hdfs.hadoopcon
各位好：
目前hadoop的配置文件是在 fs.hdfs.hadoopconf 设置。
用户是否能够不启用配置文件中的fs.hdfs.hadoopconf，通过代码手动设置hadoop的目录。
hi，
Thanks for the reply. I am using default FsStateBackend rather than rocksdb
with checkpoint off. So I really cannot see any state info from the dashboard.
I will research more details and see if any alternative can be optimized.
At 2020-01-08 19:07:08, "Benchao Li" wrote:
>hi sun
Hi
没开启Checkpoint但是想知道状态存储的用量的话，对于FsStateBackend来说没有什么好办法；但是对于RocksDBStateBackend来说可以通过开启RocksDB
native metrics [1] 的方式来观察memtable 以及 sst文件的 size，来近似估算整体状态存储数据量。
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/config.html#rocksdb-native-metrics
祝好
唐云
___
hi sunfulin,
As Kurt pointed out, if you use RocksDB state backend, maybe slow disk IO
bound your job.
You can check WindowOperator's latency metric to see how long it tasks to
process an element.
Hope this helps.
sunfulin 于2020年1月8日周三 下午4:04写道：
> Ah, I had checked resource usage and GC from fl
求问怎么通过dashboard查看状态存储量之类的统计？如果没开checkpoint的话
如果是优化器一直卡住不能退出，那基本肯定是BUG了。请开一个issue把这些信息上传上去吧，我们会调查一下是什么问题导致的。
Best,
Kurt
On Wed, Jan 8, 2020 at 5:12 PM jun su wrote:
> 添加代码文字:
>
> def main(args: Array[String]): Unit = {
>
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> env.setStreamTimeCharacteristic(TimeCharacteri
添加代码文字:
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
val tableEnv = StreamTableEnvironment.create(env)
val schema =
"{\"type\":\"record\",\"name\":\"root\",\"fiel
Ah, I had checked resource usage and GC from flink dashboard. Seem that the
reason is not cpu or memory issue. Task heap memory usage is less then 30%.
Could you kindly tell that how I can see more metrics to help target the
bottleneck?
Really appreciated that.
At 2020-01-08 15:59:17, "
hi，godfreyhe
As far as I can see, I rewrite the running sql from one count distinct level to
2 level agg, just as the table.optimizer.distinct-agg.split.enabled param
worked. Correct me if I am telling the wrong way. But the rewrite sql does not
work well for the performance throughout.
For n
10 matches
Mail list logo
