[jira] [Commented] (FLINK-9145) Website build is broken
[
https://issues.apache.org/jira/browse/FLINK-9145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428758#comment-16428758
]
Chesnay Schepler commented on FLINK-9145:
-
that is true; what I'm wondering though is why this doesn't fail on travis :/
> Website build is broken
> ---
>
> Key: FLINK-9145
> URL: https://issues.apache.org/jira/browse/FLINK-9145
> Project: Flink
> Issue Type: Bug
> Components: Project Website
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Timo Walther
>Priority: Blocker
>
> The javadoc generation fails with a dependency-convergence error in
> flink-json:
> {code}
> [WARNING]
> Dependency convergence error for commons-beanutils:commons-beanutils:1.8.0
> paths to dependency are:
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-commons-configuration:commons-configuration:1.7
> +-commons-digester:commons-digester:1.8.1
> +-commons-beanutils:commons-beanutils:1.8.0
> and
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-commons-configuration:commons-configuration:1.7
> +-commons-beanutils:commons-beanutils:1.8.3
> [WARNING]
> Dependency convergence error for org.codehaus.janino:commons-compiler:3.0.7
> paths to dependency are:
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.codehaus.janino:janino:3.0.7
> +-org.codehaus.janino:commons-compiler:3.0.7
> and
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.apache.calcite:calcite-core:1.16.0
> +-org.codehaus.janino:commons-compiler:2.7.6
> [WARNING]
> Dependency convergence error for commons-lang:commons-lang:2.6 paths to
> dependency are:
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-commons-configuration:commons-configuration:1.7
> +-commons-lang:commons-lang:2.6
> and
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.apache.calcite:calcite-core:1.16.0
> +-net.hydromatic:aggdesigner-algorithm:6.0
> +-commons-lang:commons-lang:2.4
> [WARNING]
> Dependency convergence error for org.codehaus.janino:janino:3.0.7 paths to
> dependency are:
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.codehaus.janino:janino:3.0.7
> and
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.apache.calcite:calcite-core:1.16.0
> +-org.codehaus.janino:janino:2.7.6
> and
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.codehaus.janino:janino:3.0.7
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence
> failed with message:
> Failed while enforcing releasability. See above detailed error message.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9144) Spilling batch job hangs
[
https://issues.apache.org/jira/browse/FLINK-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber updated FLINK-9144:
---
Description: A user on the mailing list reported that his batch job stops
to run with Flink 1.5 RC1:
https://lists.apache.org/thread.html/43721934405019e7255fda627afb7c9c4ed0d04fb47f1c8f346d4194@%3Cdev.flink.apache.org%3E
(was: A user on the mailing list reported that his batch job stops to run
with Flink 1.5 RC1:
https://lists.apache.org/thread.html/43721934405019e7255fda627afb7c9c4ed0d04fb47f1c8f346d4194@%3Cdev.flink.apache.org%3E
This has probably to due with {{SpilledSubpartitionView}} stopping the read
process once it reaches the end of the file and {{SpillableSubpartition#add()}}
not notifying about newly added buffers.)
> Spilling batch job hangs
>
>
> Key: FLINK-9144
> URL: https://issues.apache.org/jira/browse/FLINK-9144
> Project: Flink
> Issue Type: Bug
> Components: Network
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.5.0
>
>
> A user on the mailing list reported that his batch job stops to run with
> Flink 1.5 RC1:
> https://lists.apache.org/thread.html/43721934405019e7255fda627afb7c9c4ed0d04fb47f1c8f346d4194@%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9145) Website build is broken
[
https://issues.apache.org/jira/browse/FLINK-9145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428699#comment-16428699
]
Stephan Ewen commented on FLINK-9145:
-
The root cause is in {{flink-table}}, which should exclude janino from calcite.
> Website build is broken
> ---
>
> Key: FLINK-9145
> URL: https://issues.apache.org/jira/browse/FLINK-9145
> Project: Flink
> Issue Type: Bug
> Components: Project Website
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Timo Walther
>Priority: Blocker
>
> The javadoc generation fails with a dependency-convergence error in
> flink-json:
> {code}
> [WARNING]
> Dependency convergence error for commons-beanutils:commons-beanutils:1.8.0
> paths to dependency are:
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-commons-configuration:commons-configuration:1.7
> +-commons-digester:commons-digester:1.8.1
> +-commons-beanutils:commons-beanutils:1.8.0
> and
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-commons-configuration:commons-configuration:1.7
> +-commons-beanutils:commons-beanutils:1.8.3
> [WARNING]
> Dependency convergence error for org.codehaus.janino:commons-compiler:3.0.7
> paths to dependency are:
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.codehaus.janino:janino:3.0.7
> +-org.codehaus.janino:commons-compiler:3.0.7
> and
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.apache.calcite:calcite-core:1.16.0
> +-org.codehaus.janino:commons-compiler:2.7.6
> [WARNING]
> Dependency convergence error for commons-lang:commons-lang:2.6 paths to
> dependency are:
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-commons-configuration:commons-configuration:1.7
> +-commons-lang:commons-lang:2.6
> and
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.apache.calcite:calcite-core:1.16.0
> +-net.hydromatic:aggdesigner-algorithm:6.0
> +-commons-lang:commons-lang:2.4
> [WARNING]
> Dependency convergence error for org.codehaus.janino:janino:3.0.7 paths to
> dependency are:
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.codehaus.janino:janino:3.0.7
> and
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.apache.calcite:calcite-core:1.16.0
> +-org.codehaus.janino:janino:2.7.6
> and
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.codehaus.janino:janino:3.0.7
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence
> failed with message:
> Failed while enforcing releasability. See above detailed error message.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-6756) Provide RichAsyncFunction to Scala API suite
[ https://issues.apache.org/jira/browse/FLINK-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428676#comment-16428676 ] Elias Levy commented on FLINK-6756: --- Just ran into this, any progress? > Provide RichAsyncFunction to Scala API suite > > > Key: FLINK-6756 > URL: https://issues.apache.org/jira/browse/FLINK-6756 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Reporter: Andrea Spina >Assignee: Andrea Spina >Priority: Major > > I can't find any tracking info about the chance to have RichAsyncFunction in > the Scala API suite. I think it'd be nice to have this function in order to > access open/close methods and the RuntimeContext. > I was able to retrieve > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/There-is-no-Open-and-Close-method-in-Async-I-O-API-of-Scala-td11591.html#a11593 > only, so my question is if there are some blocking issues avoiding this > feature. [~till.rohrmann] > If it's possible and nobody already have done it, I can assign the issue to > myself in order to implement it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9144) Spilling batch job hangs
[
https://issues.apache.org/jira/browse/FLINK-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428675#comment-16428675
]
Nico Kruber commented on FLINK-9144:
It seems I have fixed the issue in
https://github.com/NicoK/flink/tree/flink-9144 but I still need to polish that
branch to create a PR.
For those affected: can you please check and verify that the job is not hanging
with Flink compiled from this branch?
> Spilling batch job hangs
>
>
> Key: FLINK-9144
> URL: https://issues.apache.org/jira/browse/FLINK-9144
> Project: Flink
> Issue Type: Bug
> Components: Network
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.5.0
>
>
> A user on the mailing list reported that his batch job stops to run with
> Flink 1.5 RC1:
> https://lists.apache.org/thread.html/43721934405019e7255fda627afb7c9c4ed0d04fb47f1c8f346d4194@%3Cdev.flink.apache.org%3E
> This has probably to due with {{SpilledSubpartitionView}} stopping the read
> process once it reaches the end of the file and
> {{SpillableSubpartition#add()}} not notifying about newly added buffers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9114) Enable user-provided, custom CheckpointRecoveryFactory for non-HA deployments
[ https://issues.apache.org/jira/browse/FLINK-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Park updated FLINK-9114: -- Description: When you operate a Flink application that uses externalized checkpoints to S3, it becomes difficult to determine which checkpoint is the latest to recover from. Because S3 provides read-after-write consistency only for PUTS, listing a S3 path is not guaranteed to be consistent, so we do not know what checkpoint to recover from. The goal of this improvement is to allow users to provide a custom CheckpointRecoveryFactory for non-HA deployments such that we can use this feature to fail checkpoints if we cannot guarantee we will know where a checkpoint will be in S3, and co-publish checkpoint metadata to a strongly consistent data store. I propose the following changes: # Modify AbstractNonHaServices and StandaloneHaServices to accept an Executor for the custom CheckpointRecoveryFactory. # Create a CheckpointRecoveryFactoryLoader to provide the custom CheckpointRecoveryFactory from configurations. # Add new configurations for this feature. We considered the pluggable StateBackend and the potentially pluggable HighAvailabilityServices. These were too convoluted to solve our problem, so we would like to implement a custom CheckpointRecoveryFactory mechanism. was: When you operate a Flink application that uses externalized checkpoints to S3, it becomes difficult to determine which checkpoint is the latest to recover from. Because S3 provides read-after-write consistency only for PUTS, listing a S3 path is not guaranteed to be consistent, so we do not know what checkpoint to recover from. The goal of this improvement is to allow users to provide a custom CheckpointRecoveryFactory for non-HA deployments such that we can use this feature to fail checkpoints if we cannot guarantee we will know where a checkpoint will be in S3, and co-publish checkpoint metadata to a strongly consistent data store. I propose the following changes: # Modify AbstractNonHaServices and StandaloneHaServices to accept an Executor for the custom CheckpointRecoveryFactory. # Create a CheckpointRecoveryFactoryLoader to provide the custom CheckpointRecoveryFactory from configurations. # Add new configurations for this feature. We considered the pluggable StateBackend and potential pluggable HighAvailabilityServices. These were too convoluted to solve our problem, so we would like custom CheckpointRecoveryFactory. > Enable user-provided, custom CheckpointRecoveryFactory for non-HA deployments > - > > Key: FLINK-9114 > URL: https://issues.apache.org/jira/browse/FLINK-9114 > Project: Flink > Issue Type: Improvement > Components: Configuration, State Backends, Checkpointing >Reporter: Jacob Park >Assignee: Jacob Park >Priority: Major > > When you operate a Flink application that uses externalized checkpoints to > S3, it becomes difficult to determine which checkpoint is the latest to > recover from. Because S3 provides read-after-write consistency only for PUTS, > listing a S3 path is not guaranteed to be consistent, so we do not know what > checkpoint to recover from. > The goal of this improvement is to allow users to provide a custom > CheckpointRecoveryFactory for non-HA deployments such that we can use this > feature to fail checkpoints if we cannot guarantee we will know where a > checkpoint will be in S3, and co-publish checkpoint metadata to a strongly > consistent data store. > I propose the following changes: > # Modify AbstractNonHaServices and StandaloneHaServices to accept an > Executor for the custom CheckpointRecoveryFactory. > # Create a CheckpointRecoveryFactoryLoader to provide the custom > CheckpointRecoveryFactory from configurations. > # Add new configurations for this feature. > We considered the pluggable StateBackend and the potentially pluggable > HighAvailabilityServices. These were too convoluted to solve our problem, so > we would like to implement a custom CheckpointRecoveryFactory mechanism. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9114) Enable user-provided, custom CheckpointRecoveryFactory for non-HA deployments
[ https://issues.apache.org/jira/browse/FLINK-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428587#comment-16428587 ] Jacob Park commented on FLINK-9114: --- I was thinking of creating a ConfigurableCheckpointRecoveryFactory interface with a configure(Configuration config, Executor executor) method instead of a constructor for ease of class-loading and reflection. It would also exist in a separate package to prevent issues with class-loading conflicts when building a JAR (like flink-metrics). The ConfigurableCheckpointRecoveryFactory will be instantiated with the help of a ConfigurableCheckpointRecoveryFactoryLoader invoked in a new abstract class that subclasses HighAvailabilityServices and overrides getCheckpointRecoveryFactory(). This new abstract class would be the parent class for YarnHighAvailabilityServices, StandaloneHaServices, and EmbeddedHaServices. I hope this approach won't be too invasive for the existing StandaloneCheckpointRecoveryFactory as the configure() method will be a no-op for it, and it would not impact how a JobManager utilizes the CheckpointRecoveryFactory to create CompletedCheckpointStore in the ExecutionGraph. > Enable user-provided, custom CheckpointRecoveryFactory for non-HA deployments > - > > Key: FLINK-9114 > URL: https://issues.apache.org/jira/browse/FLINK-9114 > Project: Flink > Issue Type: Improvement > Components: Configuration, State Backends, Checkpointing >Reporter: Jacob Park >Assignee: Jacob Park >Priority: Major > > When you operate a Flink application that uses externalized checkpoints to > S3, it becomes difficult to determine which checkpoint is the latest to > recover from. Because S3 provides read-after-write consistency only for PUTS, > listing a S3 path is not guaranteed to be consistent, so we do not know what > checkpoint to recover from. > The goal of this improvement is to allow users to provide a custom > CheckpointRecoveryFactory for non-HA deployments such that we can use this > feature to fail checkpoints if we cannot guarantee we will know where a > checkpoint will be in S3, and co-publish checkpoint metadata to a strongly > consistent data store. > I propose the following changes: > # Modify AbstractNonHaServices and StandaloneHaServices to accept an > Executor for the custom CheckpointRecoveryFactory. > # Create a CheckpointRecoveryFactoryLoader to provide the custom > CheckpointRecoveryFactory from configurations. > # Add new configurations for this feature. > We considered the pluggable StateBackend and potential pluggable > HighAvailabilityServices. These were too convoluted to solve our problem, so > we would like custom CheckpointRecoveryFactory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-7554) Add a testing RuntimeContext to test utilities
[
https://issues.apache.org/jira/browse/FLINK-7554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428541#comment-16428541
]
buptljy commented on FLINK-7554:
[~twalthr] I find that it seems to be a bit harder for testing streaming
functions like "CoProcessFunction", in which has a Context, because I can't
create a Context directly. My idea is to create a testingfunction for it, like:
{code:java}
public class TestingCoProcessFunction extends CoProcessFunction {
private TestingRuntimeContext ctx;
private Context context = new Context() {
@Override
public Long timestamp() {
return -1L;
}
@Override
public TimerService timerService() {
return new TestingTimerService();
}
@Override
public void output(OutputTag outputTag, X value) {
ctx.addSideOutput(outputTag, value);
}
};
public TestingCoProcessFunction(TestingRuntimeContext ctx) {
this.ctx = ctx;
}
public Context getContext() {
return context;
}
@Override
public void processElement1(Object value, Context ctx, Collector
out) throws Exception {}
@Override
public void processElement2(Object value, Context ctx, Collector
out) throws Exception {}
}
{code}
And the test will be like this:
{code:java}
@Test
public void testEnrichmentFunction() throws Exception {
TestingRuntimeContext ctx = new TestingRuntimeContext(true);
EnrichmentFunction func = new EnrichmentFunction();
func.setRuntimeContext(ctx);
CoProcessFunction.Context context = new
TestingCoProcessFunction(ctx).getContext();
ValueStateDescriptor rideStateDesc = new
ValueStateDescriptor<>("saved ride", TaxiRide.class);
ValueStateDescriptor fareStateDesc = new
ValueStateDescriptor<>("saved fare", TaxiFare.class);
ctx.setState(rideStateDesc, new TestingValueState(null));
ctx.setState(fareStateDesc, new TestingValueState(null));
func.open(new Configuration());
TaxiRide ride1 = new TaxiRide(1);
func.processElement1(ride1, context, ctx.getCollector());
Assert.assertEquals(ctx.getState(rideStateDesc).value(), ride1);
}
{code}
And in this way, I will create many testingfunctions for
"BroadcastProcessFunction", "CoProcessFunction"， etc. I can do this but I
wonder if you have any better ideas ?
> Add a testing RuntimeContext to test utilities
> --
>
> Key: FLINK-7554
> URL: https://issues.apache.org/jira/browse/FLINK-7554
> Project: Flink
> Issue Type: New Feature
> Components: Tests
>Reporter: Timo Walther
>Assignee: buptljy
>Priority: Major
> Labels: starter
>
> When unit testing user-defined functions it would be useful to have an
> official testing {{RuntimeContext}} that uses Java collections for storing
> state, metrics, etc.
> After executing the business logic, the user could then verify how the state
> of the UDF changed or which metrics have been collected.
> This issue includes documentation for the "Testing" section.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9114) Enable user-provided, custom CheckpointRecoveryFactory for non-HA deployments
[
https://issues.apache.org/jira/browse/FLINK-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428504#comment-16428504
]
Aljoscha Krettek commented on FLINK-9114:
-
This sounds like a good idea! Could you go a bit into details about where the
{{CheckpointRecoveryFactory}} would fit in the the lifecycle of a checkpoint?
I.e. who calls it and when, and how it will be created from a configuration in
the factory.
> Enable user-provided, custom CheckpointRecoveryFactory for non-HA deployments
> -
>
> Key: FLINK-9114
> URL: https://issues.apache.org/jira/browse/FLINK-9114
> Project: Flink
> Issue Type: Improvement
> Components: Configuration, State Backends, Checkpointing
>Reporter: Jacob Park
>Assignee: Jacob Park
>Priority: Major
>
> When you operate a Flink application that uses externalized checkpoints to
> S3, it becomes difficult to determine which checkpoint is the latest to
> recover from. Because S3 provides read-after-write consistency only for PUTS,
> listing a S3 path is not guaranteed to be consistent, so we do not know what
> checkpoint to recover from.
> The goal of this improvement is to allow users to provide a custom
> CheckpointRecoveryFactory for non-HA deployments such that we can use this
> feature to fail checkpoints if we cannot guarantee we will know where a
> checkpoint will be in S3, and co-publish checkpoint metadata to a strongly
> consistent data store.
> I propose the following changes:
> # Modify AbstractNonHaServices and StandaloneHaServices to accept an
> Executor for the custom CheckpointRecoveryFactory.
> # Create a CheckpointRecoveryFactoryLoader to provide the custom
> CheckpointRecoveryFactory from configurations.
> # Add new configurations for this feature.
> We considered the pluggable StateBackend and potential pluggable
> HighAvailabilityServices. These were too convoluted to solve our problem, so
> we would like custom CheckpointRecoveryFactory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9143) Restart strategy defined in flink-conf.yaml is ignored
[
https://issues.apache.org/jira/browse/FLINK-9143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-9143:
--
Component/s: Configuration
> Restart strategy defined in flink-conf.yaml is ignored
> --
>
> Key: FLINK-9143
> URL: https://issues.apache.org/jira/browse/FLINK-9143
> Project: Flink
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 1.4.2
>Reporter: Alex Smirnov
>Priority: Major
> Attachments: execution_config.png, jobmanager.log, jobmanager.png
>
>
> Restart strategy defined in flink-conf.yaml is disregarded, when user enables
> checkpointing.
> Steps to reproduce:
> 1. Download flink distribution (1.4.2), update flink-conf.yaml:
>
> restart-strategy: none
> state.backend: rocksdb
> state.backend.fs.checkpointdir:
> file:///tmp/nfsrecovery/flink-checkpoints-metadata
> state.backend.rocksdb.checkpointdir:
> file:///tmp/nfsrecovery/flink-checkpoints-rocksdb
>
> 2. create new java project as described at
> [https://ci.apache.org/projects/flink/flink-docs-release-1.4/quickstart/java_api_quickstart.html]
> here's the code:
> public class FailedJob
> {
> static final Logger LOGGER = LoggerFactory.getLogger(FailedJob.class);
> public static void main( String[] args ) throws Exception
> {
> final StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.enableCheckpointing(5000, CheckpointingMode.EXACTLY_ONCE);
> DataStream stream = env.fromCollection(Arrays.asList("test"));
> stream.map(new MapFunction(){
> @Override
> public String map(String obj) {
> throw new NullPointerException("NPE");
> }
> });
> env.execute("Failed job");
> }
> }
>
> 3. Compile: mvn clean package; submit it to the cluster
>
> 4. Go to Job Manager configuration in WebUI, ensure settings from
> flink-conf.yaml is there (screenshot attached)
>
> 5. Go to Job's configuration, see Execution Configuration section
>
> *Expected result*: restart strategy as defined in flink-conf.yaml
>
> *Actual result*: Restart with fixed delay (1 ms).
> #[2147483647|tel:(214)%20748-3647] restart attempts.
>
>
> see attached screenshots and jobmanager log (line 1 and 31)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-9127) Filesystem State Backend logged incorrectly
[
https://issues.apache.org/jira/browse/FLINK-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Scott Kidder closed FLINK-9127.
---
Resolution: Won't Fix
Comment from Bowen Li indicated that this log statement is in fact correct. The
statement seems non-intuitive, but oh well. Closing, won't fix.
> Filesystem State Backend logged incorrectly
> ---
>
> Key: FLINK-9127
> URL: https://issues.apache.org/jira/browse/FLINK-9127
> Project: Flink
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.3.2, 1.4.2
>Reporter: Scott Kidder
>Priority: Trivial
>
> When using a filesystem backend, the
> '[StateBackendLoader|https://github.com/apache/flink/blob/1f9c2d9740ffea2b59b8f5f3da287a0dc890ddbf/flink-runtime/src/main/java/org/apache/flink/runtime/state/StateBackendLoader.java#L123]'
> class produces a log message stating: "State backend is set to heap memory".
> Example:
> {{2018-04-04 00:45:49,591 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask - State backend
> is set to heap memory (checkpoints to filesystem
> "hdfs://hdfs:8020/flink/checkpoints")}}
> It looks like this resulted from some copy-pasta of the previous
> case-statement that matches on the memory backend. This bug is also present
> in earlier releases (1.3.2, 1.4.0) of Flink in the 'AbstractStateBackend'
> class.
> This log statement should be corrected to indicate that a filesystem backend
> is in use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9055) WebUI shows job as Running although not enough resources are available
[ https://issues.apache.org/jira/browse/FLINK-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sihua Zhou updated FLINK-9055: -- Fix Version/s: 1.6.0 > WebUI shows job as Running although not enough resources are available > -- > > Key: FLINK-9055 > URL: https://issues.apache.org/jira/browse/FLINK-9055 > Project: Flink > Issue Type: Bug > Components: JobManager, Webfrontend >Affects Versions: 1.5.0 > Environment: * FLIP-6 enabled > * Local Flink instance with fixed number of TMs > * Job parallelism exceeds available slots >Reporter: Fabian Hueske >Assignee: Sihua Zhou >Priority: Major > Fix For: 1.6.0 > > > The WebUI shows a (batch) job as "Running" although not enough resources have > been allocated to actually run the job with the requested parallelism. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9055) WebUI shows job as Running although not enough resources are available
[ https://issues.apache.org/jira/browse/FLINK-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428469#comment-16428469 ] Sihua Zhou commented on FLINK-9055: --- Ah, thanks a lot for you reply, I think Till maybe the one who know that part best, but I bet he's too busy with the working of release 1.5, I'd ping him in this issue to get some advice from him after 1.5 released out. ;) > WebUI shows job as Running although not enough resources are available > -- > > Key: FLINK-9055 > URL: https://issues.apache.org/jira/browse/FLINK-9055 > Project: Flink > Issue Type: Bug > Components: JobManager, Webfrontend >Affects Versions: 1.5.0 > Environment: * FLIP-6 enabled > * Local Flink instance with fixed number of TMs > * Job parallelism exceeds available slots >Reporter: Fabian Hueske >Assignee: Sihua Zhou >Priority: Major > Fix For: 1.6.0 > > > The WebUI shows a (batch) job as "Running" although not enough resources have > been allocated to actually run the job with the requested parallelism. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9127) Filesystem State Backend logged incorrectly
[
https://issues.apache.org/jira/browse/FLINK-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428468#comment-16428468
]
ASF GitHub Bot commented on FLINK-9127:
---
Github user skidder closed the pull request at:
https://github.com/apache/flink/pull/5810
> Filesystem State Backend logged incorrectly
> ---
>
> Key: FLINK-9127
> URL: https://issues.apache.org/jira/browse/FLINK-9127
> Project: Flink
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.3.2, 1.4.2
>Reporter: Scott Kidder
>Priority: Trivial
>
> When using a filesystem backend, the
> '[StateBackendLoader|https://github.com/apache/flink/blob/1f9c2d9740ffea2b59b8f5f3da287a0dc890ddbf/flink-runtime/src/main/java/org/apache/flink/runtime/state/StateBackendLoader.java#L123]'
> class produces a log message stating: "State backend is set to heap memory".
> Example:
> {{2018-04-04 00:45:49,591 INFO
> org.apache.flink.streaming.runtime.tasks.StreamTask - State backend
> is set to heap memory (checkpoints to filesystem
> "hdfs://hdfs:8020/flink/checkpoints")}}
> It looks like this resulted from some copy-pasta of the previous
> case-statement that matches on the memory backend. This bug is also present
> in earlier releases (1.3.2, 1.4.0) of Flink in the 'AbstractStateBackend'
> class.
> This log statement should be corrected to indicate that a filesystem backend
> is in use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #5810: [FLINK-9127] [Core] Filesystem State Backend logge...
Github user skidder closed the pull request at: https://github.com/apache/flink/pull/5810 ---
[jira] [Closed] (FLINK-8699) Fix concurrency problem in rocksdb full checkpoint
[
https://issues.apache.org/jira/browse/FLINK-8699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sihua Zhou closed FLINK-8699.
-
Resolution: Fixed
Release Note: (was: Richter has fixed this in his hotfix.)
> Fix concurrency problem in rocksdb full checkpoint
> --
>
> Key: FLINK-8699
> URL: https://issues.apache.org/jira/browse/FLINK-8699
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Blocker
> Fix For: 1.5.0
>
>
> In full checkpoint, `kvStateInformation` is not a copied object and it can be
> changed when writeKVStateMetaData() is invoking ... This can lead to
> problematic, which is serious.
> {code}
> private void writeKVStateMetaData() throws IOException {
> // ...
> for (Map.Entry RegisteredKeyedBackendStateMetaInfo>> column :
> stateBackend.kvStateInformation.entrySet()) {
> }
> //...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (FLINK-8699) Fix concurrency problem in rocksdb full checkpoint
[
https://issues.apache.org/jira/browse/FLINK-8699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sihua Zhou reopened FLINK-8699:
---
Reopen to fix the release note(The mistake that I made a long time ago...)
according to Aljoscha's comments.
> Fix concurrency problem in rocksdb full checkpoint
> --
>
> Key: FLINK-8699
> URL: https://issues.apache.org/jira/browse/FLINK-8699
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Blocker
> Fix For: 1.5.0
>
>
> In full checkpoint, `kvStateInformation` is not a copied object and it can be
> changed when writeKVStateMetaData() is invoking ... This can lead to
> problematic, which is serious.
> {code}
> private void writeKVStateMetaData() throws IOException {
> // ...
> for (Map.Entry RegisteredKeyedBackendStateMetaInfo>> column :
> stateBackend.kvStateInformation.entrySet()) {
> }
> //...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-9102) Make the JobGraph disable queued scheduling for cluster with fixed TMs
[ https://issues.apache.org/jira/browse/FLINK-9102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sihua Zhou closed FLINK-9102. - Resolution: Invalid Release Note: (was: Impossible for flip6) > Make the JobGraph disable queued scheduling for cluster with fixed TMs > -- > > Key: FLINK-9102 > URL: https://issues.apache.org/jira/browse/FLINK-9102 > Project: Flink > Issue Type: Bug > Components: Scheduler >Affects Versions: 1.5.0 >Reporter: Sihua Zhou >Assignee: Sihua Zhou >Priority: Major > Fix For: 1.5.0 > > > When we start cluster locally with fixed TMS and we should disable queued > scheduling for JobGraph. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (FLINK-9102) Make the JobGraph disable queued scheduling for cluster with fixed TMs
[ https://issues.apache.org/jira/browse/FLINK-9102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sihua Zhou reopened FLINK-9102: --- reopen for fixing release note. > Make the JobGraph disable queued scheduling for cluster with fixed TMs > -- > > Key: FLINK-9102 > URL: https://issues.apache.org/jira/browse/FLINK-9102 > Project: Flink > Issue Type: Bug > Components: Scheduler >Affects Versions: 1.5.0 >Reporter: Sihua Zhou >Assignee: Sihua Zhou >Priority: Major > Fix For: 1.5.0 > > > When we start cluster locally with fixed TMS and we should disable queued > scheduling for JobGraph. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (FLINK-9145) Website build is broken
[
https://issues.apache.org/jira/browse/FLINK-9145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther reassigned FLINK-9145:
---
Assignee: Timo Walther
> Website build is broken
> ---
>
> Key: FLINK-9145
> URL: https://issues.apache.org/jira/browse/FLINK-9145
> Project: Flink
> Issue Type: Bug
> Components: Project Website
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Timo Walther
>Priority: Blocker
>
> The javadoc generation fails with a dependency-convergence error in
> flink-json:
> {code}
> [WARNING]
> Dependency convergence error for commons-beanutils:commons-beanutils:1.8.0
> paths to dependency are:
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-commons-configuration:commons-configuration:1.7
> +-commons-digester:commons-digester:1.8.1
> +-commons-beanutils:commons-beanutils:1.8.0
> and
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-commons-configuration:commons-configuration:1.7
> +-commons-beanutils:commons-beanutils:1.8.3
> [WARNING]
> Dependency convergence error for org.codehaus.janino:commons-compiler:3.0.7
> paths to dependency are:
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.codehaus.janino:janino:3.0.7
> +-org.codehaus.janino:commons-compiler:3.0.7
> and
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.apache.calcite:calcite-core:1.16.0
> +-org.codehaus.janino:commons-compiler:2.7.6
> [WARNING]
> Dependency convergence error for commons-lang:commons-lang:2.6 paths to
> dependency are:
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-commons-configuration:commons-configuration:1.7
> +-commons-lang:commons-lang:2.6
> and
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.apache.calcite:calcite-core:1.16.0
> +-net.hydromatic:aggdesigner-algorithm:6.0
> +-commons-lang:commons-lang:2.4
> [WARNING]
> Dependency convergence error for org.codehaus.janino:janino:3.0.7 paths to
> dependency are:
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.codehaus.janino:janino:3.0.7
> and
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.apache.calcite:calcite-core:1.16.0
> +-org.codehaus.janino:janino:2.7.6
> and
> +-org.apache.flink:flink-json:1.6-SNAPSHOT
> +-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
> +-org.codehaus.janino:janino:3.0.7
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence
> failed with message:
> Failed while enforcing releasability. See above detailed error message.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9113) Data loss in BucketingSink when writing to local filesystem
[
https://issues.apache.org/jira/browse/FLINK-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther updated FLINK-9113:
Fix Version/s: 1.5.0
> Data loss in BucketingSink when writing to local filesystem
> ---
>
> Key: FLINK-9113
> URL: https://issues.apache.org/jira/browse/FLINK-9113
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Reporter: Timo Walther
>Assignee: Timo Walther
>Priority: Blocker
> Fix For: 1.5.0
>
>
> For local filesystems, it is not guaranteed that the data is flushed to disk
> during checkpointing. This leads to data loss in cases of TaskManager
> failures when writing to a local filesystem
> {{org.apache.hadoop.fs.LocalFileSystem}}. The {{flush()}} method returns a
> written length but the data is not written into the file (thus the valid
> length might be greater than the actual file size). {{hsync}} and {{hflush}}
> have no effect either.
> It seems that this behavior won't be fixed in the near future:
> https://issues.apache.org/jira/browse/HADOOP-7844
> One solution would be to call {{close()}} on a checkpoint for local
> filesystems, even though this would lead to performance decrease. If we don't
> fix this issue, we should at least add proper documentation for it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9113) Data loss in BucketingSink when writing to local filesystem
[
https://issues.apache.org/jira/browse/FLINK-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther updated FLINK-9113:
Priority: Blocker (was: Major)
> Data loss in BucketingSink when writing to local filesystem
> ---
>
> Key: FLINK-9113
> URL: https://issues.apache.org/jira/browse/FLINK-9113
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Reporter: Timo Walther
>Assignee: Timo Walther
>Priority: Blocker
> Fix For: 1.5.0
>
>
> For local filesystems, it is not guaranteed that the data is flushed to disk
> during checkpointing. This leads to data loss in cases of TaskManager
> failures when writing to a local filesystem
> {{org.apache.hadoop.fs.LocalFileSystem}}. The {{flush()}} method returns a
> written length but the data is not written into the file (thus the valid
> length might be greater than the actual file size). {{hsync}} and {{hflush}}
> have no effect either.
> It seems that this behavior won't be fixed in the near future:
> https://issues.apache.org/jira/browse/HADOOP-7844
> One solution would be to call {{close()}} on a checkpoint for local
> filesystems, even though this would lead to performance decrease. If we don't
> fix this issue, we should at least add proper documentation for it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9113) Data loss in BucketingSink when writing to local filesystem
[
https://issues.apache.org/jira/browse/FLINK-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428451#comment-16428451
]
ASF GitHub Bot commented on FLINK-9113:
---
Github user twalthr commented on the issue:
https://github.com/apache/flink/pull/5811
I will close this PR for now. Because it could not solve the general issue
mentioned in HADOOP-7844.
> Data loss in BucketingSink when writing to local filesystem
> ---
>
> Key: FLINK-9113
> URL: https://issues.apache.org/jira/browse/FLINK-9113
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Reporter: Timo Walther
>Assignee: Timo Walther
>Priority: Major
>
> For local filesystems, it is not guaranteed that the data is flushed to disk
> during checkpointing. This leads to data loss in cases of TaskManager
> failures when writing to a local filesystem
> {{org.apache.hadoop.fs.LocalFileSystem}}. The {{flush()}} method returns a
> written length but the data is not written into the file (thus the valid
> length might be greater than the actual file size). {{hsync}} and {{hflush}}
> have no effect either.
> It seems that this behavior won't be fixed in the near future:
> https://issues.apache.org/jira/browse/HADOOP-7844
> One solution would be to call {{close()}} on a checkpoint for local
> filesystems, even though this would lead to performance decrease. If we don't
> fix this issue, we should at least add proper documentation for it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9113) Data loss in BucketingSink when writing to local filesystem
[
https://issues.apache.org/jira/browse/FLINK-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428452#comment-16428452
]
ASF GitHub Bot commented on FLINK-9113:
---
Github user twalthr closed the pull request at:
https://github.com/apache/flink/pull/5811
> Data loss in BucketingSink when writing to local filesystem
> ---
>
> Key: FLINK-9113
> URL: https://issues.apache.org/jira/browse/FLINK-9113
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Reporter: Timo Walther
>Assignee: Timo Walther
>Priority: Major
>
> For local filesystems, it is not guaranteed that the data is flushed to disk
> during checkpointing. This leads to data loss in cases of TaskManager
> failures when writing to a local filesystem
> {{org.apache.hadoop.fs.LocalFileSystem}}. The {{flush()}} method returns a
> written length but the data is not written into the file (thus the valid
> length might be greater than the actual file size). {{hsync}} and {{hflush}}
> have no effect either.
> It seems that this behavior won't be fixed in the near future:
> https://issues.apache.org/jira/browse/HADOOP-7844
> One solution would be to call {{close()}} on a checkpoint for local
> filesystems, even though this would lead to performance decrease. If we don't
> fix this issue, we should at least add proper documentation for it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink issue #5811: [FLINK-9113] [connectors] Fix flushing behavior of bucket...
Github user twalthr commented on the issue: https://github.com/apache/flink/pull/5811 I will close this PR for now. Because it could not solve the general issue mentioned in HADOOP-7844. ---
[GitHub] flink pull request #5811: [FLINK-9113] [connectors] Fix flushing behavior of...
Github user twalthr closed the pull request at: https://github.com/apache/flink/pull/5811 ---
[jira] [Updated] (FLINK-9113) Data loss in BucketingSink when writing to local filesystem
[
https://issues.apache.org/jira/browse/FLINK-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther updated FLINK-9113:
Description:
For local filesystems, it is not guaranteed that the data is flushed to disk
during checkpointing. This leads to data loss in cases of TaskManager failures
when writing to a local filesystem {{org.apache.hadoop.fs.LocalFileSystem}}.
The {{flush()}} method returns a written length but the data is not written
into the file (thus the valid length might be greater than the actual file
size). {{hsync}} and {{hflush}} have no effect either.
It seems that this behavior won't be fixed in the near future:
https://issues.apache.org/jira/browse/HADOOP-7844
One solution would be to call {{close()}} on a checkpoint for local
filesystems, even though this would lead to performance decrease. If we don't
fix this issue, we should at least add proper documentation for it.
was:This issue is closely related to FLINK-7737. By default the bucketing
sink uses HDFS's {{org.apache.hadoop.fs.FSDataOutputStream#hflush}} for
performance reasons. However, this leads to data loss in case of TaskManager
failures when writing to a local filesystem
{{org.apache.hadoop.fs.LocalFileSystem}}. We should use {{hsync}} by default in
local filesystem cases and make it possible to disable this behavior if needed.
> Data loss in BucketingSink when writing to local filesystem
> ---
>
> Key: FLINK-9113
> URL: https://issues.apache.org/jira/browse/FLINK-9113
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Reporter: Timo Walther
>Assignee: Timo Walther
>Priority: Major
>
> For local filesystems, it is not guaranteed that the data is flushed to disk
> during checkpointing. This leads to data loss in cases of TaskManager
> failures when writing to a local filesystem
> {{org.apache.hadoop.fs.LocalFileSystem}}. The {{flush()}} method returns a
> written length but the data is not written into the file (thus the valid
> length might be greater than the actual file size). {{hsync}} and {{hflush}}
> have no effect either.
> It seems that this behavior won't be fixed in the near future:
> https://issues.apache.org/jira/browse/HADOOP-7844
> One solution would be to call {{close()}} on a checkpoint for local
> filesystems, even though this would lead to performance decrease. If we don't
> fix this issue, we should at least add proper documentation for it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9068) Website documentation issue - html tag visible on screen
[ https://issues.apache.org/jira/browse/FLINK-9068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428444#comment-16428444 ] ASF GitHub Bot commented on FLINK-9068: --- Github user shankarganesh1234 closed the pull request at: https://github.com/apache/flink/pull/5824 > Website documentation issue - html tag visible on screen > > > Key: FLINK-9068 > URL: https://issues.apache.org/jira/browse/FLINK-9068 > Project: Flink > Issue Type: Bug > Components: Project Website >Reporter: SHANKAR GANESH >Assignee: SHANKAR GANESH >Priority: Minor > Fix For: 1.5.0, 1.4.3 > > Attachments: Screen Shot 2018-03-23 at 7.56.48 PM.png > > > In the documentation at the following url > [https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/#physical-partitioning] > In the section which explains the 'Reduce' operator (*Reduce* > KeyedStream → DataStream), an html tag () is visible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9068) Website documentation issue - html tag visible on screen
[ https://issues.apache.org/jira/browse/FLINK-9068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428443#comment-16428443 ] ASF GitHub Bot commented on FLINK-9068: --- Github user shankarganesh1234 commented on the issue: https://github.com/apache/flink/pull/5824 @fhueske No problem. Closing PR. > Website documentation issue - html tag visible on screen > > > Key: FLINK-9068 > URL: https://issues.apache.org/jira/browse/FLINK-9068 > Project: Flink > Issue Type: Bug > Components: Project Website >Reporter: SHANKAR GANESH >Assignee: SHANKAR GANESH >Priority: Minor > Fix For: 1.5.0, 1.4.3 > > Attachments: Screen Shot 2018-03-23 at 7.56.48 PM.png > > > In the documentation at the following url > [https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/#physical-partitioning] > In the section which explains the 'Reduce' operator (*Reduce* > KeyedStream → DataStream), an html tag () is visible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] flink issue #5824: [FLINK-9068][docs] Fix for removing extra '' tag
Github user shankarganesh1234 commented on the issue: https://github.com/apache/flink/pull/5824 @fhueske No problem. Closing PR. ---
[GitHub] flink pull request #5824: [FLINK-9068][docs] Fix for removing extra '' t...
Github user shankarganesh1234 closed the pull request at: https://github.com/apache/flink/pull/5824 ---
[jira] [Commented] (FLINK-9109) Add flink modify command to documentation
[
https://issues.apache.org/jira/browse/FLINK-9109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428357#comment-16428357
]
ASF GitHub Bot commented on FLINK-9109:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5786#discussion_r179766125
--- Diff: docs/ops/cli.md ---
@@ -300,70 +306,134 @@ Action "info" shows the optimized execution plan of
the program (JSON).
program. Optional flag to override
the
default value specified in the
configuration.
- Options for yarn-cluster mode:
- -yid,--yarnapplicationIdAttach to running YARN session
-
Action "list" lists running and scheduled programs.
Syntax: list [OPTIONS]
"list" action options:
- -m,--jobmanagerAddress of the JobManager (master) to
which
- to connect. Use this flag to connect to
a
- different JobManager than the one
specified
- in the configuration.
- -r,--running Show only running programs and their
JobIDs
- -s,--scheduledShow only scheduled programs and their
JobIDs
+ -r,--running Show only running programs and their JobIDs
+ -s,--scheduled Show only scheduled programs and their JobIDs
Options for yarn-cluster mode:
+ -m,--jobmanager Address of the JobManager (master) to
+ which to connect. Use this flag to
connect
+ to a different JobManager than the
one
+ specified in the configuration.
-yid,--yarnapplicationIdAttach to running YARN session
+ -z,--zookeeperNamespace Namespace to create the Zookeeper
+ sub-paths for high availability mode
+
+ Options for flip6 mode:
+ -m,--jobmanagerAddress of the JobManager (master) to
which
+ to connect. Use this flag to connect
to a
+ different JobManager than the one
specified
+ in the configuration.
+ -z,--zookeeperNamespaceNamespace to create the Zookeeper
sub-paths
+ for high availability mode
Action "stop" stops a running program (streaming jobs only).
Syntax: stop [OPTIONS]
"stop" action options:
- -m,--jobmanagerAddress of the JobManager (master) to
which
- to connect. Use this flag to connect to
a
- different JobManager than the one
specified
- in the configuration.
+
Options for yarn-cluster mode:
+ -m,--jobmanager Address of the JobManager (master) to
+ which to connect. Use this flag to
connect
+ to a different JobManager than the
one
+ specified in the configuration.
-yid,--yarnapplicationIdAttach to running YARN session
+ -z,--zookeeperNamespace Namespace to create the Zookeeper
+ sub-paths for high availability mode
+
+ Options for flip6 mode:
--- End diff --
Good point @zentol. Have to update the PR accordingly.
> Add flink modify command to documentation
> -
>
> Key: FLINK-9109
> URL: https://issues.apache.org/jira/browse/FLINK-9109
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Blocker
> Labels: flip-6
> Fix For: 1.5.0
>
>
> We should add documentation for the {{flink modify}} command.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #5786: [FLINK-9109] [doc] Update documentation for CLI
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/5786#discussion_r179766125 --- Diff: docs/ops/cli.md --- @@ -300,70 +306,134 @@ Action "info" shows the optimized execution plan of the program (JSON). program. Optional flag to override the default value specified in the configuration. - Options for yarn-cluster mode: - -yid,--yarnapplicationIdAttach to running YARN session - Action "list" lists running and scheduled programs. Syntax: list [OPTIONS] "list" action options: - -m,--jobmanagerAddress of the JobManager (master) to which - to connect. Use this flag to connect to a - different JobManager than the one specified - in the configuration. - -r,--running Show only running programs and their JobIDs - -s,--scheduledShow only scheduled programs and their JobIDs + -r,--running Show only running programs and their JobIDs + -s,--scheduled Show only scheduled programs and their JobIDs Options for yarn-cluster mode: + -m,--jobmanager Address of the JobManager (master) to + which to connect. Use this flag to connect + to a different JobManager than the one + specified in the configuration. -yid,--yarnapplicationIdAttach to running YARN session + -z,--zookeeperNamespace Namespace to create the Zookeeper + sub-paths for high availability mode + + Options for flip6 mode: + -m,--jobmanagerAddress of the JobManager (master) to which + to connect. Use this flag to connect to a + different JobManager than the one specified + in the configuration. + -z,--zookeeperNamespaceNamespace to create the Zookeeper sub-paths + for high availability mode Action "stop" stops a running program (streaming jobs only). Syntax: stop [OPTIONS] "stop" action options: - -m,--jobmanagerAddress of the JobManager (master) to which - to connect. Use this flag to connect to a - different JobManager than the one specified - in the configuration. + Options for yarn-cluster mode: + -m,--jobmanager Address of the JobManager (master) to + which to connect. Use this flag to connect + to a different JobManager than the one + specified in the configuration. -yid,--yarnapplicationIdAttach to running YARN session + -z,--zookeeperNamespace Namespace to create the Zookeeper + sub-paths for high availability mode + + Options for flip6 mode: --- End diff -- Good point @zentol. Have to update the PR accordingly. ---
[jira] [Created] (FLINK-9145) Website build is broken
Chesnay Schepler created FLINK-9145:
---
Summary: Website build is broken
Key: FLINK-9145
URL: https://issues.apache.org/jira/browse/FLINK-9145
Project: Flink
Issue Type: Bug
Components: Project Website
Affects Versions: 1.5.0
Reporter: Chesnay Schepler
The javadoc generation fails with a dependency-convergence error in flink-json:
{code}
[WARNING]
Dependency convergence error for commons-beanutils:commons-beanutils:1.8.0
paths to dependency are:
+-org.apache.flink:flink-json:1.6-SNAPSHOT
+-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
+-commons-configuration:commons-configuration:1.7
+-commons-digester:commons-digester:1.8.1
+-commons-beanutils:commons-beanutils:1.8.0
and
+-org.apache.flink:flink-json:1.6-SNAPSHOT
+-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
+-commons-configuration:commons-configuration:1.7
+-commons-beanutils:commons-beanutils:1.8.3
[WARNING]
Dependency convergence error for org.codehaus.janino:commons-compiler:3.0.7
paths to dependency are:
+-org.apache.flink:flink-json:1.6-SNAPSHOT
+-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
+-org.codehaus.janino:janino:3.0.7
+-org.codehaus.janino:commons-compiler:3.0.7
and
+-org.apache.flink:flink-json:1.6-SNAPSHOT
+-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
+-org.apache.calcite:calcite-core:1.16.0
+-org.codehaus.janino:commons-compiler:2.7.6
[WARNING]
Dependency convergence error for commons-lang:commons-lang:2.6 paths to
dependency are:
+-org.apache.flink:flink-json:1.6-SNAPSHOT
+-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
+-commons-configuration:commons-configuration:1.7
+-commons-lang:commons-lang:2.6
and
+-org.apache.flink:flink-json:1.6-SNAPSHOT
+-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
+-org.apache.calcite:calcite-core:1.16.0
+-net.hydromatic:aggdesigner-algorithm:6.0
+-commons-lang:commons-lang:2.4
[WARNING]
Dependency convergence error for org.codehaus.janino:janino:3.0.7 paths to
dependency are:
+-org.apache.flink:flink-json:1.6-SNAPSHOT
+-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
+-org.codehaus.janino:janino:3.0.7
and
+-org.apache.flink:flink-json:1.6-SNAPSHOT
+-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
+-org.apache.calcite:calcite-core:1.16.0
+-org.codehaus.janino:janino:2.7.6
and
+-org.apache.flink:flink-json:1.6-SNAPSHOT
+-org.apache.flink:flink-table_2.11:1.6-SNAPSHOT
+-org.codehaus.janino:janino:3.0.7
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence
failed with message:
Failed while enforcing releasability. See above detailed error message.
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-8742) Move ConfigDocsGenerator annotation to flink-annotations
[
https://issues.apache.org/jira/browse/FLINK-8742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-8742.
---
Resolution: Fixed
master: 47ac3684a195eef5ce319f3e1ac3f966d98b418c
1.5: 14f0c5694ab73ba1fdb6038ce23c10f17d31d8d8
> Move ConfigDocsGenerator annotation to flink-annotations
>
>
> Key: FLINK-8742
> URL: https://issues.apache.org/jira/browse/FLINK-8742
> Project: Flink
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Labels: easy-fix, starter
> Fix For: 1.5.0
>
>
> The {{ConfigDocsGenerator}} added annotations to modify the behavior of the
> generator output. ({{ConfigGroup}} and {{ConfigGroups}}).
> Currently these annotations are part of {{flink-core}} and residein the same
> package as the various `ConfigOption` classes.
> I propose moving them to flink-annotations for consolidation. We may also
> place them in a new module in {{flink-docs}}, but given that we already have
> a module exclusively for annotations this seems overkill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-8835) Fix TaskManager config keys
[
https://issues.apache.org/jira/browse/FLINK-8835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-8835.
---
Resolution: Fixed
master: 8b0f590c52d698b3439a2c3524889802c893e985
1.5: 9e2581e0443ff47124de41a8cdcd9c18e64b0fab
> Fix TaskManager config keys
> ---
>
> Key: FLINK-8835
> URL: https://issues.apache.org/jira/browse/FLINK-8835
> Project: Flink
> Issue Type: Bug
> Components: TaskManager
>Reporter: Stephan Ewen
>Assignee: mingleizhang
>Priority: Blocker
> Labels: easy-fix
> Fix For: 1.5.0
>
>
> Many new config keys in the TaskManager don't follow the proper naming
> scheme. We need to clear those up before the release. I would also suggest to
> keep the key names short, because that makes it easier for users.
> When doing this cleanup pass over the config keys, I would suggest to also
> make some of the existing keys more hierarchical harmonize them with the
> common scheme in Flink.
> h1. New Keys
> * {{taskmanager.network.credit-based-flow-control.enabled}} to
> {{taskmanager.network.credit-model}}.
> h1. Existing Keys
> * {{taskmanager.debug.memory.startLogThread}} =>
> {{taskmanager.debug.memory.log}}
> * {{taskmanager.debug.memory.logIntervalMs}} =>
> {{taskmanager.debug.memory.log-interval}}
> * {{taskmanager.initial-registration-pause}} =>
> {{taskmanager.registration.initial-backoff}}
> * {{taskmanager.max-registration-pause}} =>
> {{taskmanager.registration.max-backoff}}
> * {{taskmanager.refused-registration-pause}}
> {{taskmanager.registration.refused-backoff}}
> * {{taskmanager.maxRegistrationDuration}} ==> *
> {{taskmanager.registration.timeout}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-8703. --- Resolution: Fixed master: 7d6afed83e370c8e36f6a690704041417e580498 1.5: 84ad2cd4b13db2dbe4a144aa2e3a2802e79f77b9 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-8837) add @Experimental annotation and properly annotate some classes
[
https://issues.apache.org/jira/browse/FLINK-8837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-8837.
---
Resolution: Fixed
master: cdd202217449cc7db7abc3c9a9c8f29f6c463c18
1.5: c8e0a314dae31b62578bab9ea4cf823b7e70b014
> add @Experimental annotation and properly annotate some classes
>
>
> Key: FLINK-8837
> URL: https://issues.apache.org/jira/browse/FLINK-8837
> Project: Flink
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 1.5.0
>Reporter: Stephan Ewen
>Assignee: Bowen Li
>Priority: Blocker
> Fix For: 1.5.0
>
>
> The class {{DataStreamUtils}} came from 'flink-contrib' and now accidentally
> moved to the fully supported API packages. It should be in package
> 'experimental' to properly communicate that it is not guaranteed to be API
> stable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-8967) Port NetworkStackThroughputITCase to flip6
[ https://issues.apache.org/jira/browse/FLINK-8967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-8967: Issue Type: Sub-task (was: Improvement) Parent: FLINK-8700 > Port NetworkStackThroughputITCase to flip6 > -- > > Key: FLINK-8967 > URL: https://issues.apache.org/jira/browse/FLINK-8967 > Project: Flink > Issue Type: Sub-task > Components: Tests >Affects Versions: 1.5.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8837) add @Experimental annotation and properly annotate some classes
[
https://issues.apache.org/jira/browse/FLINK-8837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428312#comment-16428312
]
ASF GitHub Bot commented on FLINK-8837:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/5800
> add @Experimental annotation and properly annotate some classes
>
>
> Key: FLINK-8837
> URL: https://issues.apache.org/jira/browse/FLINK-8837
> Project: Flink
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 1.5.0
>Reporter: Stephan Ewen
>Assignee: Bowen Li
>Priority: Blocker
> Fix For: 1.5.0
>
>
> The class {{DataStreamUtils}} came from 'flink-contrib' and now accidentally
> moved to the fully supported API packages. It should be in package
> 'experimental' to properly communicate that it is not guaranteed to be API
> stable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8742) Move ConfigDocsGenerator annotation to flink-annotations
[
https://issues.apache.org/jira/browse/FLINK-8742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428314#comment-16428314
]
ASF GitHub Bot commented on FLINK-8742:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/5821
> Move ConfigDocsGenerator annotation to flink-annotations
>
>
> Key: FLINK-8742
> URL: https://issues.apache.org/jira/browse/FLINK-8742
> Project: Flink
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Labels: easy-fix, starter
> Fix For: 1.5.0
>
>
> The {{ConfigDocsGenerator}} added annotations to modify the behavior of the
> generator output. ({{ConfigGroup}} and {{ConfigGroups}}).
> Currently these annotations are part of {{flink-core}} and residein the same
> package as the various `ConfigOption` classes.
> I propose moving them to flink-annotations for consolidation. We may also
> place them in a new module in {{flink-docs}}, but given that we already have
> a module exclusively for annotations this seems overkill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428311#comment-16428311 ] ASF GitHub Bot commented on FLINK-8703: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5669 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8835) Fix TaskManager config keys
[
https://issues.apache.org/jira/browse/FLINK-8835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428313#comment-16428313
]
ASF GitHub Bot commented on FLINK-8835:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/5808
> Fix TaskManager config keys
> ---
>
> Key: FLINK-8835
> URL: https://issues.apache.org/jira/browse/FLINK-8835
> Project: Flink
> Issue Type: Bug
> Components: TaskManager
>Reporter: Stephan Ewen
>Assignee: mingleizhang
>Priority: Blocker
> Labels: easy-fix
> Fix For: 1.5.0
>
>
> Many new config keys in the TaskManager don't follow the proper naming
> scheme. We need to clear those up before the release. I would also suggest to
> keep the key names short, because that makes it easier for users.
> When doing this cleanup pass over the config keys, I would suggest to also
> make some of the existing keys more hierarchical harmonize them with the
> common scheme in Flink.
> h1. New Keys
> * {{taskmanager.network.credit-based-flow-control.enabled}} to
> {{taskmanager.network.credit-model}}.
> h1. Existing Keys
> * {{taskmanager.debug.memory.startLogThread}} =>
> {{taskmanager.debug.memory.log}}
> * {{taskmanager.debug.memory.logIntervalMs}} =>
> {{taskmanager.debug.memory.log-interval}}
> * {{taskmanager.initial-registration-pause}} =>
> {{taskmanager.registration.initial-backoff}}
> * {{taskmanager.max-registration-pause}} =>
> {{taskmanager.registration.max-backoff}}
> * {{taskmanager.refused-registration-pause}}
> {{taskmanager.registration.refused-backoff}}
> * {{taskmanager.maxRegistrationDuration}} ==> *
> {{taskmanager.registration.timeout}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #5808: [FLINK-8835] [taskmanager] Fix TaskManager config ...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5808 ---
[GitHub] flink pull request #5760: [hotfix] [doc] update maven versions in building f...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5760 ---
[GitHub] flink pull request #5669: [FLINK-8703][tests] Port KafkaTestBase to MiniClus...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5669 ---
[GitHub] flink pull request #5800: [FLINK-8837] [DataStream API] add @Experimental an...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5800 ---
[GitHub] flink pull request #5821: [FLINK-8742][docs] Move docs generator annotations...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5821 ---
[GitHub] flink pull request #5787: Release 1.5
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5787 ---
[jira] [Commented] (FLINK-9113) Data loss in BucketingSink when writing to local filesystem
[
https://issues.apache.org/jira/browse/FLINK-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428297#comment-16428297
]
ASF GitHub Bot commented on FLINK-9113:
---
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5811#discussion_r179743870
--- Diff:
flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
---
@@ -366,6 +384,11 @@ public void
initializeState(FunctionInitializationContext context) throws Except
throw new RuntimeException("Error while creating
FileSystem when initializing the state of the BucketingSink.", e);
}
+ // sync on flush for local file systems
+ if (localSyncOnFlush && (fs instanceof LocalFileSystem) &&
(writerTemplate instanceof StreamWriterBase)) {
--- End diff --
Shouldn't `(writerTemplate instanceof StreamWriterBase)` check be converted
here into `checState(writerTemplate instanceof StreamWriterBase)` inside the if
branch, and the same check be extracted and validated whenever user calls:
`setWriter(...)` or `setLocalSyncOnFlush(...)`? So that non
`StreamWriterBase` and `localSyncOnFlush = true` would be invalid
configuration? Otherwise users might experience `wtf` moments when flag is
being ignored after changing their writer.
> Data loss in BucketingSink when writing to local filesystem
> ---
>
> Key: FLINK-9113
> URL: https://issues.apache.org/jira/browse/FLINK-9113
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Reporter: Timo Walther
>Assignee: Timo Walther
>Priority: Major
>
> This issue is closely related to FLINK-7737. By default the bucketing sink
> uses HDFS's {{org.apache.hadoop.fs.FSDataOutputStream#hflush}} for
> performance reasons. However, this leads to data loss in case of TaskManager
> failures when writing to a local filesystem
> {{org.apache.hadoop.fs.LocalFileSystem}}. We should use {{hsync}} by default
> in local filesystem cases and make it possible to disable this behavior if
> needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #5811: [FLINK-9113] [connectors] Fix flushing behavior of...
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5811#discussion_r179743870
--- Diff:
flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
---
@@ -366,6 +384,11 @@ public void
initializeState(FunctionInitializationContext context) throws Except
throw new RuntimeException("Error while creating
FileSystem when initializing the state of the BucketingSink.", e);
}
+ // sync on flush for local file systems
+ if (localSyncOnFlush && (fs instanceof LocalFileSystem) &&
(writerTemplate instanceof StreamWriterBase)) {
--- End diff --
Shouldn't `(writerTemplate instanceof StreamWriterBase)` check be converted
here into `checState(writerTemplate instanceof StreamWriterBase)` inside the if
branch, and the same check be extracted and validated whenever user calls:
`setWriter(...)` or `setLocalSyncOnFlush(...)`? So that non
`StreamWriterBase` and `localSyncOnFlush = true` would be invalid
configuration? Otherwise users might experience `wtf` moments when flag is
being ignored after changing their writer.
---
[jira] [Commented] (FLINK-9113) Data loss in BucketingSink when writing to local filesystem
[
https://issues.apache.org/jira/browse/FLINK-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428287#comment-16428287
]
ASF GitHub Bot commented on FLINK-9113:
---
Github user twalthr commented on the issue:
https://github.com/apache/flink/pull/5811
Thanks for looking into it @kl0u. I observed the same behavior during
debugging. I will remove the check for now and open a follow up issue. If there
is no better solution, we might need to close the writer for checkpoints on
local filesystems for preventing data loss in cases where the OS/machine goes
down.
> Data loss in BucketingSink when writing to local filesystem
> ---
>
> Key: FLINK-9113
> URL: https://issues.apache.org/jira/browse/FLINK-9113
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Reporter: Timo Walther
>Assignee: Timo Walther
>Priority: Major
>
> This issue is closely related to FLINK-7737. By default the bucketing sink
> uses HDFS's {{org.apache.hadoop.fs.FSDataOutputStream#hflush}} for
> performance reasons. However, this leads to data loss in case of TaskManager
> failures when writing to a local filesystem
> {{org.apache.hadoop.fs.LocalFileSystem}}. We should use {{hsync}} by default
> in local filesystem cases and make it possible to disable this behavior if
> needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink issue #5811: [FLINK-9113] [connectors] Fix flushing behavior of bucket...
Github user twalthr commented on the issue: https://github.com/apache/flink/pull/5811 Thanks for looking into it @kl0u. I observed the same behavior during debugging. I will remove the check for now and open a follow up issue. If there is no better solution, we might need to close the writer for checkpoints on local filesystems for preventing data loss in cases where the OS/machine goes down. ---
[jira] [Commented] (FLINK-9113) Data loss in BucketingSink when writing to local filesystem
[
https://issues.apache.org/jira/browse/FLINK-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428266#comment-16428266
]
ASF GitHub Bot commented on FLINK-9113:
---
Github user kl0u commented on the issue:
https://github.com/apache/flink/pull/5811
Well it seems like for these tests, the `flush` is not actually flushing.
The files are there, the `validPartLength` is correct (=6 as we just write
`test1\n`) but the data is not actually on disk. If you call `close()` on the
in-progress file when snapshotting, then the tests succeed and the data is
there.
I would recommend to just remove the check for now, and open a followup
JIRA that contains the check that you will remove, and also points on the
discussion about HDFS not flushing, and we see how to proceed.
I thing that the fact that the end-to-end tests pass point to the direction
that sth is wrong with the FS abstraction.
> Data loss in BucketingSink when writing to local filesystem
> ---
>
> Key: FLINK-9113
> URL: https://issues.apache.org/jira/browse/FLINK-9113
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Reporter: Timo Walther
>Assignee: Timo Walther
>Priority: Major
>
> This issue is closely related to FLINK-7737. By default the bucketing sink
> uses HDFS's {{org.apache.hadoop.fs.FSDataOutputStream#hflush}} for
> performance reasons. However, this leads to data loss in case of TaskManager
> failures when writing to a local filesystem
> {{org.apache.hadoop.fs.LocalFileSystem}}. We should use {{hsync}} by default
> in local filesystem cases and make it possible to disable this behavior if
> needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink issue #5811: [FLINK-9113] [connectors] Fix flushing behavior of bucket...
Github user kl0u commented on the issue: https://github.com/apache/flink/pull/5811 Well it seems like for these tests, the `flush` is not actually flushing. The files are there, the `validPartLength` is correct (=6 as we just write `test1\n`) but the data is not actually on disk. If you call `close()` on the in-progress file when snapshotting, then the tests succeed and the data is there. I would recommend to just remove the check for now, and open a followup JIRA that contains the check that you will remove, and also points on the discussion about HDFS not flushing, and we see how to proceed. I thing that the fact that the end-to-end tests pass point to the direction that sth is wrong with the FS abstraction. ---
[jira] [Assigned] (FLINK-9005) Cluster test: Run general purpose job with failures with Mesos
[
https://issues.apache.org/jira/browse/FLINK-9005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao reassigned FLINK-9005:
---
Assignee: Gary Yao
> Cluster test: Run general purpose job with failures with Mesos
> --
>
> Key: FLINK-9005
> URL: https://issues.apache.org/jira/browse/FLINK-9005
> Project: Flink
> Issue Type: Sub-task
> Components: Mesos, Tests
>Reporter: Till Rohrmann
>Assignee: Gary Yao
>Priority: Critical
>
> Similar to FLINK-9004, we should run the general purpose job (FLINK-8971)
> with failures on Mesos.
> We should use the Hadoop-free Flink version and write our checkpoints to HDFS
> with Flink's HDFS file system.
> We should use the {{FsStateBackend}} with asynchronous checkpoints for the
> test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-2685) TaskManager deadlock on NetworkBufferPool
[
https://issues.apache.org/jira/browse/FLINK-2685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber updated FLINK-2685:
---
Component/s: Network
> TaskManager deadlock on NetworkBufferPool
> -
>
> Key: FLINK-2685
> URL: https://issues.apache.org/jira/browse/FLINK-2685
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination, Network
>Affects Versions: 0.10.0
>Reporter: Greg Hogan
>Assignee: Ufuk Celebi
>Priority: Major
> Attachments: job_manager_19_feb_15_30_running,
> task_manager_19_feb_15_30_running
>
>
> This deadlock occurs intermittently. I have a {{join}} followed by a
> {{chain}} followed by a {{reduceGroup}}. Stack traces and local
> variables from one each of the {{join}} threads below.
> The {{join}}'s are waiting on a buffer to become available
> ({{networkBufferPool.availableMemorySegments.count=0}}). Both
> {{LocalBufferPool}}'s have been given extra capacity ({{currentPoolSize=60 >
> numberOfRequiredMemorySegments=32}}). The first {{join}} is at full capacity
> ({{currentPoolSize=numberOfRequestedMemorySegments=60}}) yet the second
> {{join}} has not acquired any ({{numberOfRequestedMemorySegments=0}}).
> {{LocalBufferPool.returnExcessMemorySegments}} only recycles
> {{MemorySegment}}'s from its {{availableMemorySegments}}, so any requested
> {{Buffer}}'s will only be released when explicitly recycled.
> First join stack trace and variable values from
> {{LocalBufferPool.requestBuffer}}:
> {noformat}
> owns: SpanningRecordSerializer (id=723)
> waiting for: ArrayDeque (id=724)
> Object.wait(long) line: not available [native method]
> LocalBufferPool.requestBuffer(boolean) line: 163
> LocalBufferPool.requestBufferBlocking() line: 133
> RecordWriter.emit(T) line: 92
> OutputCollector.collect(T) line: 65
> JoinOperator$ProjectFlatJoinFunction.join(T1, T2, Collector)
> line: 1088
> ReusingBuildSecondHashMatchIterator.callWithNextKey(FlatJoinFunction,
> Collector) line: 137
> JoinDriver.run() line: 208
> RegularPactTask.run() line: 489
> RegularPactTask.invoke() line: 354
> Task.run() line: 581
> Thread.run() line: 745
> {noformat}
> {noformat}
> this LocalBufferPool (id=403)
> availableMemorySegments ArrayDeque (id=398)
> elementsObject[16] (id=422)
> head14
> tail14
> currentPoolSize 60
> isDestroyed false
> networkBufferPool NetworkBufferPool (id=354)
> allBufferPools HashSet (id=424)
> availableMemorySegments ArrayBlockingQueue (id=427)
> count 0
> items Object[10240] (id=674)
> itrsnull
> lockReentrantLock (id=675)
> notEmpty
> AbstractQueuedSynchronizer$ConditionObject (id=678)
> notFull AbstractQueuedSynchronizer$ConditionObject
> (id=679)
> putIndex6954
> takeIndex 6954
> factoryLock Object (id=430)
> isDestroyed false
> managedBufferPools HashSet (id=431)
> memorySegmentSize 32768
> numTotalRequiredBuffers 3226
> totalNumberOfMemorySegments 10240
> numberOfRequestedMemorySegments 60
> numberOfRequiredMemorySegments 32
> owner null
> registeredListeners ArrayDeque (id=421)
> elementsObject[16] (id=685)
> head0
> tail0
> askToRecycle false
> isBlockingtrue
> {noformat}
> Second join stack trace and variable values from
> {{SingleInputGate.getNextBufferOrEvent}}:
> {noformat}
> Unsafe.park(boolean, long) line: not available [native method]
> LockSupport.parkNanos(Object, long) line: 215
> AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) line: 2078
> LinkedBlockingQueue.poll(long, TimeUnit) line: 467
> SingleInputGate.getNextBufferOrEvent() line: 414
> MutableRecordReader(AbstractRecordReader).getNextRecord(T) line: 79
> MutableRecordReader.next(T) line: 34
> ReaderIterator.next(T) line: 59
> MutableHashTable$ProbeIterator.next() line: 1581
> MutableHashTable.processProbeIter() line: 457
> MutableHashTable.nextRecord() line: 555
> ReusingBuildSecondHashMatchIterator.callWithNextKey(FlatJoinFunction,
> Collector) line: 110
>
[jira] [Resolved] (FLINK-9059) Add support for unified table source and sink declaration in environment file
[ https://issues.apache.org/jira/browse/FLINK-9059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther resolved FLINK-9059. - Resolution: Fixed Fixed in 1.6.0: bc9982c364f54c40223d14eeb4823a882c021e7a Fixed in 1.5.0: ff3a0ec692b1314caccbcb482618d4229e130a9b > Add support for unified table source and sink declaration in environment file > - > > Key: FLINK-9059 > URL: https://issues.apache.org/jira/browse/FLINK-9059 > Project: Flink > Issue Type: Task > Components: Table API SQL >Reporter: Shuyi Chen >Assignee: Shuyi Chen >Priority: Major > Fix For: 1.5.0 > > > 1) Add a common property called "type" with single value 'source'. > 2) in yaml file, replace "sources" with "tables". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9059) Add support for unified table source and sink declaration in environment file
[ https://issues.apache.org/jira/browse/FLINK-9059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428247#comment-16428247 ] ASF GitHub Bot commented on FLINK-9059: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5758 > Add support for unified table source and sink declaration in environment file > - > > Key: FLINK-9059 > URL: https://issues.apache.org/jira/browse/FLINK-9059 > Project: Flink > Issue Type: Task > Components: Table API SQL >Reporter: Shuyi Chen >Assignee: Shuyi Chen >Priority: Major > Fix For: 1.5.0 > > > 1) Add a common property called "type" with single value 'source'. > 2) in yaml file, replace "sources" with "tables". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] flink pull request #5758: [FLINK-9059][Table API & SQL] add table type attri...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5758 ---
[jira] [Closed] (FLINK-8968) Fix native resource leak caused by ReadOptions
[
https://issues.apache.org/jira/browse/FLINK-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Richter closed FLINK-8968.
-
Resolution: Fixed
Merged in f5071d7a61
> Fix native resource leak caused by ReadOptions
> ---
>
> Key: FLINK-8968
> URL: https://issues.apache.org/jira/browse/FLINK-8968
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Blocker
> Fix For: 1.5.0
>
>
> We should pull the creation of ReadOptions out of the loop in
> {{RocksDBFullSnapshotOperation.writeKVStateMetaData()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-8699) Fix concurrency problem in rocksdb full checkpoint
[
https://issues.apache.org/jira/browse/FLINK-8699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Richter closed FLINK-8699.
-
Resolution: Fixed
Merged in 21cf59d5ff
> Fix concurrency problem in rocksdb full checkpoint
> --
>
> Key: FLINK-8699
> URL: https://issues.apache.org/jira/browse/FLINK-8699
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Blocker
> Fix For: 1.5.0
>
>
> In full checkpoint, `kvStateInformation` is not a copied object and it can be
> changed when writeKVStateMetaData() is invoking ... This can lead to
> problematic, which is serious.
> {code}
> private void writeKVStateMetaData() throws IOException {
> // ...
> for (Map.Entry RegisteredKeyedBackendStateMetaInfo>> column :
> stateBackend.kvStateInformation.entrySet()) {
> }
> //...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8699) Fix concurrency problem in rocksdb full checkpoint
[
https://issues.apache.org/jira/browse/FLINK-8699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428182#comment-16428182
]
ASF GitHub Bot commented on FLINK-8699:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/5705
> Fix concurrency problem in rocksdb full checkpoint
> --
>
> Key: FLINK-8699
> URL: https://issues.apache.org/jira/browse/FLINK-8699
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Blocker
> Fix For: 1.5.0
>
>
> In full checkpoint, `kvStateInformation` is not a copied object and it can be
> changed when writeKVStateMetaData() is invoking ... This can lead to
> problematic, which is serious.
> {code}
> private void writeKVStateMetaData() throws IOException {
> // ...
> for (Map.Entry RegisteredKeyedBackendStateMetaInfo>> column :
> stateBackend.kvStateInformation.entrySet()) {
> }
> //...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #5705: [FLINK-8699][FLINK-8968][state]Fix native resource...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5705 ---
[jira] [Commented] (FLINK-9068) Website documentation issue - html tag visible on screen
[ https://issues.apache.org/jira/browse/FLINK-9068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428178#comment-16428178 ] ASF GitHub Bot commented on FLINK-9068: --- Github user fhueske commented on the issue: https://github.com/apache/flink/pull/5824 Hi @shankarganesh1234, I forgot to close the PR while merging. Can you please close it? Thank you, Fabian > Website documentation issue - html tag visible on screen > > > Key: FLINK-9068 > URL: https://issues.apache.org/jira/browse/FLINK-9068 > Project: Flink > Issue Type: Bug > Components: Project Website >Reporter: SHANKAR GANESH >Assignee: SHANKAR GANESH >Priority: Minor > Fix For: 1.5.0, 1.4.3 > > Attachments: Screen Shot 2018-03-23 at 7.56.48 PM.png > > > In the documentation at the following url > [https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/#physical-partitioning] > In the section which explains the 'Reduce' operator (*Reduce* > KeyedStream → DataStream), an html tag () is visible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] flink issue #5824: [FLINK-9068][docs] Fix for removing extra '' tag
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/5824 Hi @shankarganesh1234, I forgot to close the PR while merging. Can you please close it? Thank you, Fabian ---
[GitHub] flink pull request #5786: [FLINK-9109] [doc] Update documentation for CLI
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/5786#discussion_r179716924 --- Diff: docs/ops/cli.md --- @@ -300,70 +306,134 @@ Action "info" shows the optimized execution plan of the program (JSON). program. Optional flag to override the default value specified in the configuration. - Options for yarn-cluster mode: - -yid,--yarnapplicationIdAttach to running YARN session - Action "list" lists running and scheduled programs. Syntax: list [OPTIONS] "list" action options: - -m,--jobmanagerAddress of the JobManager (master) to which - to connect. Use this flag to connect to a - different JobManager than the one specified - in the configuration. - -r,--running Show only running programs and their JobIDs - -s,--scheduledShow only scheduled programs and their JobIDs + -r,--running Show only running programs and their JobIDs + -s,--scheduled Show only scheduled programs and their JobIDs Options for yarn-cluster mode: + -m,--jobmanager Address of the JobManager (master) to + which to connect. Use this flag to connect + to a different JobManager than the one + specified in the configuration. -yid,--yarnapplicationIdAttach to running YARN session + -z,--zookeeperNamespace Namespace to create the Zookeeper + sub-paths for high availability mode + + Options for flip6 mode: + -m,--jobmanagerAddress of the JobManager (master) to which + to connect. Use this flag to connect to a + different JobManager than the one specified + in the configuration. + -z,--zookeeperNamespaceNamespace to create the Zookeeper sub-paths + for high availability mode Action "stop" stops a running program (streaming jobs only). Syntax: stop [OPTIONS] "stop" action options: - -m,--jobmanagerAddress of the JobManager (master) to which - to connect. Use this flag to connect to a - different JobManager than the one specified - in the configuration. + Options for yarn-cluster mode: + -m,--jobmanager Address of the JobManager (master) to + which to connect. Use this flag to connect + to a different JobManager than the one + specified in the configuration. -yid,--yarnapplicationIdAttach to running YARN session + -z,--zookeeperNamespace Namespace to create the Zookeeper + sub-paths for high availability mode + + Options for flip6 mode: --- End diff -- You recently removed flip6 references from classes, do we retain them in the documentation? ---
[jira] [Commented] (FLINK-9109) Add flink modify command to documentation
[
https://issues.apache.org/jira/browse/FLINK-9109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428177#comment-16428177
]
ASF GitHub Bot commented on FLINK-9109:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5786#discussion_r179716924
--- Diff: docs/ops/cli.md ---
@@ -300,70 +306,134 @@ Action "info" shows the optimized execution plan of
the program (JSON).
program. Optional flag to override
the
default value specified in the
configuration.
- Options for yarn-cluster mode:
- -yid,--yarnapplicationIdAttach to running YARN session
-
Action "list" lists running and scheduled programs.
Syntax: list [OPTIONS]
"list" action options:
- -m,--jobmanagerAddress of the JobManager (master) to
which
- to connect. Use this flag to connect to
a
- different JobManager than the one
specified
- in the configuration.
- -r,--running Show only running programs and their
JobIDs
- -s,--scheduledShow only scheduled programs and their
JobIDs
+ -r,--running Show only running programs and their JobIDs
+ -s,--scheduled Show only scheduled programs and their JobIDs
Options for yarn-cluster mode:
+ -m,--jobmanager Address of the JobManager (master) to
+ which to connect. Use this flag to
connect
+ to a different JobManager than the
one
+ specified in the configuration.
-yid,--yarnapplicationIdAttach to running YARN session
+ -z,--zookeeperNamespace Namespace to create the Zookeeper
+ sub-paths for high availability mode
+
+ Options for flip6 mode:
+ -m,--jobmanagerAddress of the JobManager (master) to
which
+ to connect. Use this flag to connect
to a
+ different JobManager than the one
specified
+ in the configuration.
+ -z,--zookeeperNamespaceNamespace to create the Zookeeper
sub-paths
+ for high availability mode
Action "stop" stops a running program (streaming jobs only).
Syntax: stop [OPTIONS]
"stop" action options:
- -m,--jobmanagerAddress of the JobManager (master) to
which
- to connect. Use this flag to connect to
a
- different JobManager than the one
specified
- in the configuration.
+
Options for yarn-cluster mode:
+ -m,--jobmanager Address of the JobManager (master) to
+ which to connect. Use this flag to
connect
+ to a different JobManager than the
one
+ specified in the configuration.
-yid,--yarnapplicationIdAttach to running YARN session
+ -z,--zookeeperNamespace Namespace to create the Zookeeper
+ sub-paths for high availability mode
+
+ Options for flip6 mode:
--- End diff --
You recently removed flip6 references from classes, do we retain them in
the documentation?
> Add flink modify command to documentation
> -
>
> Key: FLINK-9109
> URL: https://issues.apache.org/jira/browse/FLINK-9109
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Blocker
> Labels: flip-6
> Fix For: 1.5.0
>
>
> We should add documentation for the {{flink modify}} command.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink issue #5760: [hotfix] [doc] update maven versions in building flink pa...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/5760 merging. ---
[jira] [Closed] (FLINK-9068) Website documentation issue - html tag visible on screen
[ https://issues.apache.org/jira/browse/FLINK-9068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske closed FLINK-9068. Resolution: Fixed Fix Version/s: 1.4.3 1.5.0 Fixed for 1.6.0 with 2d872447df517ca63ff05f0ff264a71725eb68bd Fixed for 1.5.0 with d3395e02e7bbf77deda107e1486761688945e631 Fixed for 1.4.3 with bfe3ad5fe4ab2827547c7442776918f5f4c50ffd > Website documentation issue - html tag visible on screen > > > Key: FLINK-9068 > URL: https://issues.apache.org/jira/browse/FLINK-9068 > Project: Flink > Issue Type: Bug > Components: Project Website >Reporter: SHANKAR GANESH >Assignee: SHANKAR GANESH >Priority: Minor > Fix For: 1.5.0, 1.4.3 > > Attachments: Screen Shot 2018-03-23 at 7.56.48 PM.png > > > In the documentation at the following url > [https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/#physical-partitioning] > In the section which explains the 'Reduce' operator (*Reduce* > KeyedStream → DataStream), an html tag () is visible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8742) Move ConfigDocsGenerator annotation to flink-annotations
[
https://issues.apache.org/jira/browse/FLINK-8742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428173#comment-16428173
]
ASF GitHub Bot commented on FLINK-8742:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/5821
merging.
> Move ConfigDocsGenerator annotation to flink-annotations
>
>
> Key: FLINK-8742
> URL: https://issues.apache.org/jira/browse/FLINK-8742
> Project: Flink
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Labels: easy-fix, starter
> Fix For: 1.5.0
>
>
> The {{ConfigDocsGenerator}} added annotations to modify the behavior of the
> generator output. ({{ConfigGroup}} and {{ConfigGroups}}).
> Currently these annotations are part of {{flink-core}} and residein the same
> package as the various `ConfigOption` classes.
> I propose moving them to flink-annotations for consolidation. We may also
> place them in a new module in {{flink-docs}}, but given that we already have
> a module exclusively for annotations this seems overkill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink issue #5821: [FLINK-8742][docs] Move docs generator annotations to fli...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/5821 merging. ---
[GitHub] flink pull request #5758: [FLINK-9059][Table API & SQL] add table type attri...
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/5758#discussion_r179715295
--- Diff:
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/config/Environment.java
---
@@ -29,38 +30,47 @@
/**
* Environment configuration that represents the content of an environment
file. Environment files
- * define sources, execution, and deployment behavior. An environment
might be defined by default or
+ * define tables, execution, and deployment behavior. An environment might
be defined by default or
* as part of a session. Environments can be merged or enriched with
properties (e.g. from CLI command).
*
* In future versions, we might restrict the merging or enrichment of
deployment properties to not
* allow overwriting of a deployment by a session.
*/
public class Environment {
- private Map sources;
+ private Map tables;
--- End diff --
Also true. I will merge this now to have it in the next release. We can
still refactor if necessary as it is internal API.
---
[jira] [Commented] (FLINK-9059) Add support for unified table source and sink declaration in environment file
[
https://issues.apache.org/jira/browse/FLINK-9059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428168#comment-16428168
]
ASF GitHub Bot commented on FLINK-9059:
---
Github user twalthr commented on a diff in the pull request:
https://github.com/apache/flink/pull/5758#discussion_r179715295
--- Diff:
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/config/Environment.java
---
@@ -29,38 +30,47 @@
/**
* Environment configuration that represents the content of an environment
file. Environment files
- * define sources, execution, and deployment behavior. An environment
might be defined by default or
+ * define tables, execution, and deployment behavior. An environment might
be defined by default or
* as part of a session. Environments can be merged or enriched with
properties (e.g. from CLI command).
*
* In future versions, we might restrict the merging or enrichment of
deployment properties to not
* allow overwriting of a deployment by a session.
*/
public class Environment {
- private Map sources;
+ private Map tables;
--- End diff --
Also true. I will merge this now to have it in the next release. We can
still refactor if necessary as it is internal API.
> Add support for unified table source and sink declaration in environment file
> -
>
> Key: FLINK-9059
> URL: https://issues.apache.org/jira/browse/FLINK-9059
> Project: Flink
> Issue Type: Task
> Components: Table API SQL
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
> Fix For: 1.5.0
>
>
> 1) Add a common property called "type" with single value 'source'.
> 2) in yaml file, replace "sources" with "tables".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8835) Fix TaskManager config keys
[
https://issues.apache.org/jira/browse/FLINK-8835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428165#comment-16428165
]
ASF GitHub Bot commented on FLINK-8835:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/5808
merging.
> Fix TaskManager config keys
> ---
>
> Key: FLINK-8835
> URL: https://issues.apache.org/jira/browse/FLINK-8835
> Project: Flink
> Issue Type: Bug
> Components: TaskManager
>Reporter: Stephan Ewen
>Assignee: mingleizhang
>Priority: Blocker
> Labels: easy-fix
> Fix For: 1.5.0
>
>
> Many new config keys in the TaskManager don't follow the proper naming
> scheme. We need to clear those up before the release. I would also suggest to
> keep the key names short, because that makes it easier for users.
> When doing this cleanup pass over the config keys, I would suggest to also
> make some of the existing keys more hierarchical harmonize them with the
> common scheme in Flink.
> h1. New Keys
> * {{taskmanager.network.credit-based-flow-control.enabled}} to
> {{taskmanager.network.credit-model}}.
> h1. Existing Keys
> * {{taskmanager.debug.memory.startLogThread}} =>
> {{taskmanager.debug.memory.log}}
> * {{taskmanager.debug.memory.logIntervalMs}} =>
> {{taskmanager.debug.memory.log-interval}}
> * {{taskmanager.initial-registration-pause}} =>
> {{taskmanager.registration.initial-backoff}}
> * {{taskmanager.max-registration-pause}} =>
> {{taskmanager.registration.max-backoff}}
> * {{taskmanager.refused-registration-pause}}
> {{taskmanager.registration.refused-backoff}}
> * {{taskmanager.maxRegistrationDuration}} ==> *
> {{taskmanager.registration.timeout}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink issue #5808: [FLINK-8835] [taskmanager] Fix TaskManager config keys
Github user zentol commented on the issue: https://github.com/apache/flink/pull/5808 merging. ---
[GitHub] flink issue #5824: [FLINK-9068][docs] Fix for removing extra '' tag
Github user zentol commented on the issue: https://github.com/apache/flink/pull/5824 merging. ---
[jira] [Commented] (FLINK-9068) Website documentation issue - html tag visible on screen
[ https://issues.apache.org/jira/browse/FLINK-9068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428160#comment-16428160 ] ASF GitHub Bot commented on FLINK-9068: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5824 merging. > Website documentation issue - html tag visible on screen > > > Key: FLINK-9068 > URL: https://issues.apache.org/jira/browse/FLINK-9068 > Project: Flink > Issue Type: Bug > Components: Project Website >Reporter: SHANKAR GANESH >Assignee: SHANKAR GANESH >Priority: Minor > Attachments: Screen Shot 2018-03-23 at 7.56.48 PM.png > > > In the documentation at the following url > [https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/#physical-partitioning] > In the section which explains the 'Reduce' operator (*Reduce* > KeyedStream → DataStream), an html tag () is visible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8837) add @Experimental annotation and properly annotate some classes
[
https://issues.apache.org/jira/browse/FLINK-8837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428158#comment-16428158
]
ASF GitHub Bot commented on FLINK-8837:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/5800
merging.
> add @Experimental annotation and properly annotate some classes
>
>
> Key: FLINK-8837
> URL: https://issues.apache.org/jira/browse/FLINK-8837
> Project: Flink
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 1.5.0
>Reporter: Stephan Ewen
>Assignee: Bowen Li
>Priority: Blocker
> Fix For: 1.5.0
>
>
> The class {{DataStreamUtils}} came from 'flink-contrib' and now accidentally
> moved to the fully supported API packages. It should be in package
> 'experimental' to properly communicate that it is not guaranteed to be API
> stable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink issue #5800: [FLINK-8837] [DataStream API] add @Experimental annotatio...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/5800 merging. ---
[jira] [Commented] (FLINK-9068) Website documentation issue - html tag visible on screen
[ https://issues.apache.org/jira/browse/FLINK-9068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428155#comment-16428155 ] ASF GitHub Bot commented on FLINK-9068: --- Github user fhueske commented on the issue: https://github.com/apache/flink/pull/5824 Thanks for the PR @shankarganesh1234. +1 to merge > Website documentation issue - html tag visible on screen > > > Key: FLINK-9068 > URL: https://issues.apache.org/jira/browse/FLINK-9068 > Project: Flink > Issue Type: Bug > Components: Project Website >Reporter: SHANKAR GANESH >Assignee: SHANKAR GANESH >Priority: Minor > Attachments: Screen Shot 2018-03-23 at 7.56.48 PM.png > > > In the documentation at the following url > [https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/#physical-partitioning] > In the section which explains the 'Reduce' operator (*Reduce* > KeyedStream → DataStream), an html tag () is visible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] flink issue #5824: [FLINK-9068][docs] Fix for removing extra '' tag
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/5824 Thanks for the PR @shankarganesh1234. +1 to merge ---
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428152#comment-16428152 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5669 merging. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] flink issue #5669: [FLINK-8703][tests] Port KafkaTestBase to MiniClusterReso...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/5669 merging. ---
[jira] [Created] (FLINK-9144) Spilling batch job hangs
Nico Kruber created FLINK-9144:
--
Summary: Spilling batch job hangs
Key: FLINK-9144
URL: https://issues.apache.org/jira/browse/FLINK-9144
Project: Flink
Issue Type: Bug
Components: Network
Affects Versions: 1.5.0, 1.6.0
Reporter: Nico Kruber
Assignee: Nico Kruber
Fix For: 1.5.0
A user on the mailing list reported that his batch job stops to run with Flink
1.5 RC1:
https://lists.apache.org/thread.html/43721934405019e7255fda627afb7c9c4ed0d04fb47f1c8f346d4194@%3Cdev.flink.apache.org%3E
This has probably to due with {{SpilledSubpartitionView}} stopping the read
process once it reaches the end of the file and {{SpillableSubpartition#add()}}
not notifying about newly added buffers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9059) Add support for unified table source and sink declaration in environment file
[
https://issues.apache.org/jira/browse/FLINK-9059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428081#comment-16428081
]
ASF GitHub Bot commented on FLINK-9059:
---
Github user suez1224 commented on a diff in the pull request:
https://github.com/apache/flink/pull/5758#discussion_r179686471
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/descriptors/TableDescriptor.scala
---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.descriptors
+
+import org.apache.flink.table.descriptors.DescriptorProperties.toScala
+import
org.apache.flink.table.descriptors.StatisticsValidator.{STATISTICS_COLUMNS,
STATISTICS_ROW_COUNT, readColumnStats}
+import org.apache.flink.table.plan.stats.TableStats
+
+import scala.collection.JavaConverters._
+
+/**
+ * Common class for all descriptors describing table sources and sinks.
+ */
+abstract class TableDescriptor extends Descriptor {
+
+ protected var connectorDescriptor: Option[ConnectorDescriptor] = None
+ protected var formatDescriptor: Option[FormatDescriptor] = None
+ protected var schemaDescriptor: Option[Schema] = None
+ protected var statisticsDescriptor: Option[Statistics] = None
--- End diff --
Thanks a lot, Timo. We can consider refactoring the code later. How about
simply moving statisticsDescriptor to SourceTableDescriptor for now?
> Add support for unified table source and sink declaration in environment file
> -
>
> Key: FLINK-9059
> URL: https://issues.apache.org/jira/browse/FLINK-9059
> Project: Flink
> Issue Type: Task
> Components: Table API SQL
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
> Fix For: 1.5.0
>
>
> 1) Add a common property called "type" with single value 'source'.
> 2) in yaml file, replace "sources" with "tables".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9059) Add support for unified table source and sink declaration in environment file
[
https://issues.apache.org/jira/browse/FLINK-9059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428080#comment-16428080
]
ASF GitHub Bot commented on FLINK-9059:
---
Github user suez1224 commented on a diff in the pull request:
https://github.com/apache/flink/pull/5758#discussion_r179686465
--- Diff:
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/config/Environment.java
---
@@ -29,38 +30,47 @@
/**
* Environment configuration that represents the content of an environment
file. Environment files
- * define sources, execution, and deployment behavior. An environment
might be defined by default or
+ * define tables, execution, and deployment behavior. An environment might
be defined by default or
* as part of a session. Environments can be merged or enriched with
properties (e.g. from CLI command).
*
* In future versions, we might restrict the merging or enrichment of
deployment properties to not
* allow overwriting of a deployment by a session.
*/
public class Environment {
- private Map sources;
+ private Map tables;
--- End diff --
That is another option. But the purpose is if a table is both source and
sink, we dont need to duplicate the config in both the sources and sinks
section, as it might be error prone that you modify the table config in
sources, but forget to modify the config of the same table in sinks section,
thus causing inconsistency. What do you think?
> Add support for unified table source and sink declaration in environment file
> -
>
> Key: FLINK-9059
> URL: https://issues.apache.org/jira/browse/FLINK-9059
> Project: Flink
> Issue Type: Task
> Components: Table API SQL
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
> Fix For: 1.5.0
>
>
> 1) Add a common property called "type" with single value 'source'.
> 2) in yaml file, replace "sources" with "tables".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] flink pull request #5758: [FLINK-9059][Table API & SQL] add table type attri...
Github user suez1224 commented on a diff in the pull request:
https://github.com/apache/flink/pull/5758#discussion_r179686465
--- Diff:
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/config/Environment.java
---
@@ -29,38 +30,47 @@
/**
* Environment configuration that represents the content of an environment
file. Environment files
- * define sources, execution, and deployment behavior. An environment
might be defined by default or
+ * define tables, execution, and deployment behavior. An environment might
be defined by default or
* as part of a session. Environments can be merged or enriched with
properties (e.g. from CLI command).
*
* In future versions, we might restrict the merging or enrichment of
deployment properties to not
* allow overwriting of a deployment by a session.
*/
public class Environment {
- private Map sources;
+ private Map tables;
--- End diff --
That is another option. But the purpose is if a table is both source and
sink, we dont need to duplicate the config in both the sources and sinks
section, as it might be error prone that you modify the table config in
sources, but forget to modify the config of the same table in sinks section,
thus causing inconsistency. What do you think?
---
[GitHub] flink pull request #5758: [FLINK-9059][Table API & SQL] add table type attri...
Github user suez1224 commented on a diff in the pull request:
https://github.com/apache/flink/pull/5758#discussion_r179686471
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/descriptors/TableDescriptor.scala
---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.descriptors
+
+import org.apache.flink.table.descriptors.DescriptorProperties.toScala
+import
org.apache.flink.table.descriptors.StatisticsValidator.{STATISTICS_COLUMNS,
STATISTICS_ROW_COUNT, readColumnStats}
+import org.apache.flink.table.plan.stats.TableStats
+
+import scala.collection.JavaConverters._
+
+/**
+ * Common class for all descriptors describing table sources and sinks.
+ */
+abstract class TableDescriptor extends Descriptor {
+
+ protected var connectorDescriptor: Option[ConnectorDescriptor] = None
+ protected var formatDescriptor: Option[FormatDescriptor] = None
+ protected var schemaDescriptor: Option[Schema] = None
+ protected var statisticsDescriptor: Option[Statistics] = None
--- End diff --
Thanks a lot, Timo. We can consider refactoring the code later. How about
simply moving statisticsDescriptor to SourceTableDescriptor for now?
---
[jira] [Commented] (FLINK-9143) Restart strategy defined in flink-conf.yaml is ignored
[
https://issues.apache.org/jira/browse/FLINK-9143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428060#comment-16428060
]
Alex Smirnov commented on FLINK-9143:
-
mailing list discusssion:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Restart-strategy-defined-in-flink-conf-yaml-is-ignored-td19361.html
> Restart strategy defined in flink-conf.yaml is ignored
> --
>
> Key: FLINK-9143
> URL: https://issues.apache.org/jira/browse/FLINK-9143
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.4.2
>Reporter: Alex Smirnov
>Priority: Major
> Attachments: execution_config.png, jobmanager.log, jobmanager.png
>
>
> Restart strategy defined in flink-conf.yaml is disregarded, when user enables
> checkpointing.
> Steps to reproduce:
> 1. Download flink distribution (1.4.2), update flink-conf.yaml:
>
> restart-strategy: none
> state.backend: rocksdb
> state.backend.fs.checkpointdir:
> file:///tmp/nfsrecovery/flink-checkpoints-metadata
> state.backend.rocksdb.checkpointdir:
> file:///tmp/nfsrecovery/flink-checkpoints-rocksdb
>
> 2. create new java project as described at
> [https://ci.apache.org/projects/flink/flink-docs-release-1.4/quickstart/java_api_quickstart.html]
> here's the code:
> public class FailedJob
> {
> static final Logger LOGGER = LoggerFactory.getLogger(FailedJob.class);
> public static void main( String[] args ) throws Exception
> {
> final StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.enableCheckpointing(5000, CheckpointingMode.EXACTLY_ONCE);
> DataStream stream = env.fromCollection(Arrays.asList("test"));
> stream.map(new MapFunction(){
> @Override
> public String map(String obj) {
> throw new NullPointerException("NPE");
> }
> });
> env.execute("Failed job");
> }
> }
>
> 3. Compile: mvn clean package; submit it to the cluster
>
> 4. Go to Job Manager configuration in WebUI, ensure settings from
> flink-conf.yaml is there (screenshot attached)
>
> 5. Go to Job's configuration, see Execution Configuration section
>
> *Expected result*: restart strategy as defined in flink-conf.yaml
>
> *Actual result*: Restart with fixed delay (1 ms).
> #[2147483647|tel:(214)%20748-3647] restart attempts.
>
>
> see attached screenshots and jobmanager log (line 1 and 31)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-9143) Restart strategy defined in flink-conf.yaml is ignored
Alex Smirnov created FLINK-9143:
---
Summary: Restart strategy defined in flink-conf.yaml is ignored
Key: FLINK-9143
URL: https://issues.apache.org/jira/browse/FLINK-9143
Project: Flink
Issue Type: Bug
Affects Versions: 1.4.2
Reporter: Alex Smirnov
Attachments: execution_config.png, jobmanager.log, jobmanager.png
Restart strategy defined in flink-conf.yaml is disregarded, when user enables
checkpointing.
Steps to reproduce:
1. Download flink distribution (1.4.2), update flink-conf.yaml:
restart-strategy: none
state.backend: rocksdb
state.backend.fs.checkpointdir:
file:///tmp/nfsrecovery/flink-checkpoints-metadata
state.backend.rocksdb.checkpointdir:
file:///tmp/nfsrecovery/flink-checkpoints-rocksdb
2. create new java project as described at
[https://ci.apache.org/projects/flink/flink-docs-release-1.4/quickstart/java_api_quickstart.html]
here's the code:
public class FailedJob
{
static final Logger LOGGER = LoggerFactory.getLogger(FailedJob.class);
public static void main( String[] args ) throws Exception
{
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(5000, CheckpointingMode.EXACTLY_ONCE);
DataStream stream = env.fromCollection(Arrays.asList("test"));
stream.map(new MapFunction(){
@Override
public String map(String obj) {
throw new NullPointerException("NPE");
}
});
env.execute("Failed job");
}
}
3. Compile: mvn clean package; submit it to the cluster
4. Go to Job Manager configuration in WebUI, ensure settings from
flink-conf.yaml is there (screenshot attached)
5. Go to Job's configuration, see Execution Configuration section
*Expected result*: restart strategy as defined in flink-conf.yaml
*Actual result*: Restart with fixed delay (1 ms).
#[2147483647|tel:(214)%20748-3647] restart attempts.
see attached screenshots and jobmanager log (line 1 and 31)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8335) Upgrade hbase connector dependency to 1.4.3
[ https://issues.apache.org/jira/browse/FLINK-8335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428004#comment-16428004 ] ASF GitHub Bot commented on FLINK-8335: --- Github user zhangminglei commented on the issue: https://github.com/apache/flink/pull/5488 Hi, @StephanEwen and @greghogan I think this patch is ready for having another review now. Could you please ? Thank you. And the CI error does not relevant to this issue itself. > Upgrade hbase connector dependency to 1.4.3 > --- > > Key: FLINK-8335 > URL: https://issues.apache.org/jira/browse/FLINK-8335 > Project: Flink > Issue Type: Improvement > Components: Batch Connectors and Input/Output Formats >Reporter: Ted Yu >Assignee: mingleizhang >Priority: Minor > > hbase 1.4.1 has been released. > 1.4.0 shows speed improvement over previous 1.x releases. > http://search-hadoop.com/m/HBase/YGbbBxedD1Mnm8t?subj=Re+VOTE+The+second+HBase+1+4+0+release+candidate+RC1+is+available > This issue is to upgrade the dependency to 1.4.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] flink issue #5488: [FLINK-8335] [hbase] Upgrade hbase connector dependency t...
Github user zhangminglei commented on the issue: https://github.com/apache/flink/pull/5488 Hi, @StephanEwen and @greghogan I think this patch is ready for having another review now. Could you please ? Thank you. And the CI error does not relevant to this issue itself. ---
