Re: Flink disaster recovery test problems
Hi Zhong, Looks you are assigning tasks to different slot sharing groups to force them to not share the same slot. So you will need at least 2 slots for the streaming job to start running successfully. Killing one of the 2 TM, one slot in each, will lead to insufficient slots and your job will hang at slot allocation. Task states are needed to not skip unprocessed source data, thus to avoid data loss. It's also needed if you want the failed task to recovery to the state right before failure. Checkpointing is needed to persist the task states. If it is not enabled, the job will restart with the initial state, i.e. the job will consume data from the very beginning and there can be a big data regression. Thanks, Zhu Zhu 钟旭阳 于2019年11月5日周二 下午3:01写道： > hello: > > > I am currently learning flink.I recently had a problem with Flink for > disaster recovery testing.I tried to find an answer on the official website > and blog but failed.I am trying to find community help. > > > The current situation is：I have two servers, each with one slot.My > application has two parallel operators with a degree of parallelism of 1, > using the slotSharingGroup function to make them run in these two slots > respectively. > > > My disaster recovery test is to shut down one of the servers. But is it > possible that two parallel operators compete for the same server slot? In > addition to this,I want to dynamically add or remove servers (simulated > power failures,etc) while Flink is running, but I think this must cause > stream data loss. Is it only one way to restart Flink through the > checkpoint mechanism to ensure that data is not lost and the number of > servers is dynamically configured? > > > Best > Zhong
Re: Job Distribution Strategy On Cluster.
Hi srikanth, What's your job's parallelism? In some scenes, many operators are chained with each other. if it's parallelism is 1, it would just use a single slot. Best, Vino srikanth flink 于2019年11月6日周三 下午10:03写道： > Hi there, > > I'm running Flink with 3 node cluster. > While running my jobs(both SQL client and jar submission), the jobs are > being assigned to single machine instead of distribution among the cluster. > How could I achieve the job distribution to make use of the computation > power? > > Thanks > Srikanth >
Re: Job Distribution Strategy On Cluster.
Hi Srikanth, Is this issue what you encounter? FLINK-12122: a job would tend to fill one TM before using another. If it is, you may need to wait for the release 1.9.2 or 1.10, since it is just fixed. Thanks, Zhu Zhu vino yang 于2019年11月11日周一 下午5:48写道： > Hi srikanth, > > What's your job's parallelism? > > In some scenes, many operators are chained with each other. if it's > parallelism is 1, it would just use a single slot. > > Best, > Vino > > srikanth flink 于2019年11月6日周三 下午10:03写道： > >> Hi there, >> >> I'm running Flink with 3 node cluster. >> While running my jobs(both SQL client and jar submission), the jobs are >> being assigned to single machine instead of distribution among the cluster. >> How could I achieve the job distribution to make use of the computation >> power? >> >> Thanks >> Srikanth >> >
Document an example pattern that makes sources and sinks pluggable in the production code for testing
Hi guys, I found the testing part mentioned make sources and sinks pluggable in your production code and inject special test sources and test sinks in your tests. https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#testing-flink-jobs I think it would be useful to have a documented example as the section *testing stateful operato*r does, which demonstrates by WindowOperatorTest https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#unit-testing-stateful-or-timely-udfs--custom-operators or, is there perhaps already a test that plugs sources and sinks? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Document an example pattern that makes sources and sinks pluggable in the production code for testing
Hi guys, I found the testing part mentioned make sources and sinks pluggable in your production code and inject special test sources and test sinks in your tests. https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#testing-flink-jobs I think it would be useful to have a documented example as the section *testing stateful operato*r does, which demonstrates by WindowOperatorTest https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#unit-testing-stateful-or-timely-udfs--custom-operators or, is there perhaps already a test that plugs sources and sinks? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Job Distribution Strategy On Cluster.
Vina, I've set parallelism as 6 while max parallelism is 128. Thanks Srikanth On Mon, Nov 11, 2019 at 3:18 PM vino yang wrote: > Hi srikanth, > > What's your job's parallelism? > > In some scenes, many operators are chained with each other. if it's > parallelism is 1, it would just use a single slot. > > Best, > Vino > > srikanth flink 于2019年11月6日周三 下午10:03写道： > >> Hi there, >> >> I'm running Flink with 3 node cluster. >> While running my jobs(both SQL client and jar submission), the jobs are >> being assigned to single machine instead of distribution among the cluster. >> How could I achieve the job distribution to make use of the computation >> power? >> >> Thanks >> Srikanth >> >
Re: Job Distribution Strategy On Cluster.
Zhu Zhu, That's awesome and is what I'm looking for. Any update on when would be the next release date? Thanks Srikanth On Mon, Nov 11, 2019 at 3:40 PM Zhu Zhu wrote: > Hi Srikanth, > > Is this issue what you encounter? FLINK-12122: a job would tend to fill > one TM before using another. > If it is, you may need to wait for the release 1.9.2 or 1.10, since it is > just fixed. > > Thanks, > Zhu Zhu > > vino yang 于2019年11月11日周一 下午5:48写道： > >> Hi srikanth, >> >> What's your job's parallelism? >> >> In some scenes, many operators are chained with each other. if it's >> parallelism is 1, it would just use a single slot. >> >> Best, >> Vino >> >> srikanth flink 于2019年11月6日周三 下午10:03写道： >> >>> Hi there, >>> >>> I'm running Flink with 3 node cluster. >>> While running my jobs(both SQL client and jar submission), the jobs are >>> being assigned to single machine instead of distribution among the cluster. >>> How could I achieve the job distribution to make use of the computation >>> power? >>> >>> Thanks >>> Srikanth >>> >>
Propagating event time field from nested query
Hi folks, We have a Flink streaming Table / SQL job that we were looking to migrate from an older Flink release (1.6.x) to 1.9. As part of doing so, we have been seeing a few errors which I was trying to figure out how to work around. Would appreciate any help / pointers. Job essentially involves a nested query: SELECT `timestamp`, cost, partnerid, impression_id, … FROM my_kafka_stream The kafka stream has a ‘timestamp’ field that tracks event time. We register this nested query as “base_query”. We now use this in a couple of outer aggregation queries (different outer aggregation queries differ in terms of the time window we aggregate over – 1M, 1H, 6H etc): SELECT SUM(cost) AS FLOAT AS CostPerPartner, COUNT(impression_id) AS ImpsPerPartner, … FROM base_query GROUP BY partnerid, HOP(`timestamp`, INTERVAL '30' SECOND, INTERVAL '1' MINUTE) While the outer query would get translated and scheduled as a Flink streaming job just fine on 1.6, we are running into this error when we try to bump our build to 1.9: “Exception in thread "main" org.apache.flink.table.api.ValidationException: Window can only be defined over a time attribute column.” Any suggestions on how we could work around this? I saw a thread suggesting using HOP_ROWTIME but if I understand correctly, that would mean we would need to do the hop window generation / group by in the nested query which we’d like to avoid (as we have a couple of time window combinations to generate). Thanks, -- Piyush
Re: Monitor rocksDB memory usage
Thanks! I'll check it out. Best Lu On Thu, Nov 7, 2019 at 10:24 PM Yun Tang wrote: > Hi Lu > > > > I think RocksDB native metrics [1] could help. > > > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/config.html#rocksdb-native-metrics > > > > Best > > Yun Tang > > > > *From: *Lu Niu > *Date: *Friday, November 8, 2019 at 8:18 AM > *To: *user > *Subject: *Monitor rocksDB memory usage > > > > Hi, > > > > I read that rocksDB memory is managed off heap. Is there a way to monitor > the memory usage there then? > > > > Best > > Lu >
Re: Document an example pattern that makes sources and sinks pluggable in the production code for testing
Hi Hung, Your suggestion is reasonable. Giving an example of a pluggable source and sink can make it more user-friendly, you can open a JIRA issue to see if there is anyone who wants to improve this. IMO, it's not very difficult to implement it. Because the source and sink in Flink has two unified abstract interfaces: `SourceFunction` and `SinkFunction`. You can pass them into your method which builds your DAG(except source and sink). Then, you can provide different implementation of source and sink for production and testing purposes. Best, Vino Hung 于2019年11月11日周一 下午8:44写道： > Hi guys, > > I found the testing part mentioned > > make sources and sinks pluggable in your production code and inject special > test sources and test sinks in your tests. > > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#testing-flink-jobs > > I think it would be useful to have a documented example as the section > *testing stateful operato*r does, which demonstrates by WindowOperatorTest > > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#unit-testing-stateful-or-timely-udfs--custom-operators > > or, is there perhaps already a test that plugs sources and sinks? > > > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >
Re: Job Distribution Strategy On Cluster.
There is no plan for release 1.9.2 yet. Flink 1.10.0 is planned to be released in early January. Thanks, Zhu Zhu srikanth flink 于2019年11月11日周一 下午9:53写道： > Zhu Zhu, > > That's awesome and is what I'm looking for. > Any update on when would be the next release date? > > Thanks > Srikanth > > On Mon, Nov 11, 2019 at 3:40 PM Zhu Zhu wrote: > >> Hi Srikanth, >> >> Is this issue what you encounter? FLINK-12122: a job would tend to fill >> one TM before using another. >> If it is, you may need to wait for the release 1.9.2 or 1.10, since it is >> just fixed. >> >> Thanks, >> Zhu Zhu >> >> vino yang 于2019年11月11日周一 下午5:48写道： >> >>> Hi srikanth, >>> >>> What's your job's parallelism? >>> >>> In some scenes, many operators are chained with each other. if it's >>> parallelism is 1, it would just use a single slot. >>> >>> Best, >>> Vino >>> >>> srikanth flink 于2019年11月6日周三 下午10:03写道： >>> Hi there, I'm running Flink with 3 node cluster. While running my jobs(both SQL client and jar submission), the jobs are being assigned to single machine instead of distribution among the cluster. How could I achieve the job distribution to make use of the computation power? Thanks Srikanth >>>
Re: Job Distribution Strategy On Cluster.
Great, thanks for the update. On Tue, Nov 12, 2019 at 8:51 AM Zhu Zhu wrote: > There is no plan for release 1.9.2 yet. > Flink 1.10.0 is planned to be released in early January. > > Thanks, > Zhu Zhu > > srikanth flink 于2019年11月11日周一 下午9:53写道： > >> Zhu Zhu, >> >> That's awesome and is what I'm looking for. >> Any update on when would be the next release date? >> >> Thanks >> Srikanth >> >> On Mon, Nov 11, 2019 at 3:40 PM Zhu Zhu wrote: >> >>> Hi Srikanth, >>> >>> Is this issue what you encounter? FLINK-12122: a job would tend to fill >>> one TM before using another. >>> If it is, you may need to wait for the release 1.9.2 or 1.10, since it >>> is just fixed. >>> >>> Thanks, >>> Zhu Zhu >>> >>> vino yang 于2019年11月11日周一 下午5:48写道： >>> Hi srikanth, What's your job's parallelism? In some scenes, many operators are chained with each other. if it's parallelism is 1, it would just use a single slot. Best, Vino srikanth flink 于2019年11月6日周三 下午10:03写道： > Hi there, > > I'm running Flink with 3 node cluster. > While running my jobs(both SQL client and jar submission), the jobs > are being assigned to single machine instead of distribution among the > cluster. How could I achieve the job distribution to make use of the > computation power? > > Thanks > Srikanth >
Re: static table in flink
Hi Jaqie: I think you can take a look to temporal table in blink planner. (Using LookupableTableSource). [1] Use processing time: Its processing is to query from external table(storage like HBase, JDBC) by per record in real time. [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/joins.html#join-with-a-temporal-table Best, Jingsong Lee On Sun, Nov 10, 2019 at 11:00 AM vino yang wrote: > Hi Jaqie, > > If I understand your question correctly, it seems you are finding a > solution about the Stream table and Dim table(you called static table) join. > > There were many users who asked this question. Linked some reply > here[1][2] to let you consider. > > Best, > Vino > > [1]: > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/How-to-Join-a-dimension-table-in-flink-sql-td20729.html > [2]: http://www.codeha.us/apache-flink-users/msg12421.html > > Jaqie Chan 于2019年11月9日周六 下午3:04写道： > >> Hello, >> >> I have questions about static table in flink. >> >> Join the stream table with static table. I'm looking at temporal table, >> while the time based table would grow exponentially over period. Any >> suggestions? >> >> Stream tables checks the contains in static table(updates once everyday >> with new set of data). Trying to approach this with views. >> >> Thanks for any suggestions. >> >> Regards >> 嘉琪 >> >> -- Best, Jingsong Lee
