Hello,
My suggestion would be to try testing with Docker locally (maybe using a
specific tag even) to see if you can replicate the problem.
Doing a `docker run --rm -p 8081:8081 flink:latest local` just now I am
able to access the Flink dashboard as expected.
If you're deploying on Kubernetes, th
got it.
thanks guys
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
One explanation would be that during catch up, data is consumer with higher
throughput because its just read from Kafka.
Hence, you'd see also more late data per minute while the job catches up
until it reads data at the rate at which it is produced into Kafka.
Would that explain your observations
Hi,
The release notes state that "multiple slots are not *fully* supported".
In Flink 1.5.0, the configured number of slots is ignored when requesting
containers for TaskManagers from a resource manager, i.e., Flink assumes
TMs with 1 slot.
Hence, Flink request too many containers and starts too
Hi Kien!
Thank you for sharing your experience. Do you have Flink deployed in
Standalone Cluster mode or K8S/Docker mode?
On Fri, 1 Jun 2018 at 00:55, Fabian Hueske wrote:
> Hi,
>
> The release notes state that "multiple slots are not *fully* supported".
>
> In Flink 1.5.0, the configured numbe
FYI: https://issues.apache.org/jira/browse/FLINK-9455 tracks the problem
with the multi slot support for the SlotManager.
Cheers,
Till
On Fri, Jun 1, 2018 at 10:10 AM, Abdul Qadeer wrote:
> Hi Kien!
>
> Thank you for sharing your experience. Do you have Flink deployed in
> Standalone Cluster mo
Hi,
have you tried exporting `HADOOP_USER_NAME` with the hbase user before
running your application?
Cheers,
Till
On Fri, Jun 1, 2018 at 6:31 AM, Xinyu Zhang wrote:
> Hi all
>
> I'm trying to write data to HDFS in directory "/home/hbase/XXX". Only
> "hbase" user can write to the directory.
> T
A heads up on this front:
- For state backends during checkpointing, I would suggest to use the
flink-s3-fs-presto, which is quite a bit faster than the flink-s3-fs-hadoop
by avoiding a bunch of unnecessary metadata operations.
- We have started work on re-writing the Bucketing Sink to make i
I am running these Flink job in a Yarn cluster. When the flink job fails or I
restart it I want the Flink job to use the savepoint that was created before
it restarted. How do can restart the Flink job with this savepoint. I am
using Flink 1.4.2.
--
Sent from: http://apache-flink-user-mailing-li
Hi Bajaj,
this is currently not possible. Could you please open a JIRA issue for this
feature to be added. I think in the end the change should be not resetting
the CREATED timestamp when calling `ExecutionGraph#restart()`.
Cheers,
Till
On Thu, May 31, 2018 at 6:34 PM, Bajaj, Abhinav
wrote:
>
Hi Miki,
could you check whether the files are really no longer stored on HDFS? How
did you terminate the cluster? Simply calling `bin/stop-cluster.sh`? I just
tried it locally and it could recover the job after calling
`bin/start-cluster.sh` again.
What would be helpful are the logs from the ini
Hi,
the reason why `jobmanager.execution.failover-strategy` defaults to `full`
is that `individual` only works if you have a completely embarrassingly
parallel job where all operators chain into a single task.
Concerning `taskmanager.jvm-exit-on-oom`, depending on where the
OOMException has been
Hi Anil,
when submitting the job, simply add `bin/flink run --fromSavepoint
-p `.
Cheers,
Till
On Fri, Jun 1, 2018 at 4:43 PM, Anil wrote:
> I am running these Flink job in a Yarn cluster. When the flink job fails
> or I
> restart it I want the Flink job to use the savepoint that was created
Hi,
Could anyone help me to solve this problem
/Exception in thread "main" java.lang.Error: Unresolved compilation problem:
The constructor Kafka09JsonTableSource(String, Properties,
TypeInformation) is undefined
/
*--This is the code *
public class FlinkKafkaSQL {
public stat
Hi everybody,
Due to popular demand, we've extended the Call for Presentations for Flink
Forward Berlin 2018 by one week.
The call will close on *Monday, June 11* (11:59pm CEST).
Please submit a proposal to present your Flink and Stream Processing use
case, experiences, and best practices in Berl
Hi Xinyu,
To add to Till's comment, setting `HADOOP_USER_NAME` in your environment is
probably the easiest way if you are using CLI.
If you are launching the job programmatically, e.g. using
YarnClusterDescriptor [1], there're many ways to set `HADOOP_USER_NAME` as
well, please share more informat
Hi Anil,
Glad to know that you upgrade the system to 1.4, from our experience there
are quite a bit of changes requires to adapt to the new deployment model in
1.4 if I remember correctly.
The Deployment model "run detach" in AthenaX does not support reattach back
to the job, we use REST API to do
Yeah that's my observation too. Basically small chunks of late data can get
added up quickly when data is read at a faster rate.
On a related note, I would expect if there is no late data produced in
Kafka, then immaterial of what rate the data is read, this problem should
not occur.
To take care
Thank you for creating this Till, so as I understand this won't be
supported (for standalone cluster mode) even if legacy mode is enabled?
On Fri, Jun 1, 2018 at 6:21 AM, Till Rohrmann wrote:
> FYI: https://issues.apache.org/jira/browse/FLINK-9455 tracks the problem
> with the multi slot support
For standalone mode it works for the legacy and the new mode. Only when
using Yarn or Mesos, it can happen that you temporarily allocate too many
container. The unused container will, however, be freed if they idle too
long.
Cheers,
Till
On Fri, Jun 1, 2018, 21:52 Abdul Qadeer wrote:
> Thank yo
Thanks Georgi,
I ended up chucking the POM example from AWS ref architecture and re-generating
a new pom from Flink 1.4 archetype, and then logging into a master node via SSH
and submitting a job directly. Using Steps on EMR Console doesn’t seem to be
quite the same thing I guess.
Thanks for y
Hi Radhya,
Can you provide which Flink version you are using? Based on the latest
FLINK 1.5 release, Kafka09JsonTableSource takes:
/**
* Creates a Kafka 0.9 JSON {@link StreamTableSource}.
*
* @param topic Kafka topic to consume.
* @param properties Properties for the Kafka consumer.
I can not find anywhere I have 100M. Not sure why I get this failure.
This is in my dev docker env. Same configure file worked well for 1.3.2
= Log
Caused by: org.apache.flink.util.FlinkException: Failed to submit job
aa75905062dd0487034bb9d8b6617dc2.
at
org.apache.flink.runtime
Hi,
I have a use case where I am keeping the keyed state in ProcessFunction.
Key: Integer personId;
/**
* The data type stored in the state
*/public class PersonDetails {
public long count;
public long lastModified;}
I have encountered a lot of ConcurrentModificationException.
I tho
Hi Garvit,
this is unexpected, could you please provide more information about this?
- which flink version are you using?
- what state backend are you using?
- are you using the incremental checkpoint?(in case you used the rocksdb
backend)
- did you create the customer thread to operate the sta
I am trying to figure out how to use S3 as state storage.
The recommended way is
https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/aws.html#shaded-hadooppresto-s3-file-systems-recommended
Seems like I only have to do two things:
*1. Put flink-s3-fs-presto to the lib*
*2. C
26 matches
Mail list logo
