[jira] [Created] (FLINK-32025) Make job cancellation button on UI configurable
Ted Yu created FLINK-32025: -- Summary: Make job cancellation button on UI configurable Key: FLINK-32025 URL: https://issues.apache.org/jira/browse/FLINK-32025 Project: Flink Issue Type: Improvement Reporter: Ted Yu On the flink job UI, there is `Cancel Job` button. When the job UI is shown to users, it is desirable to hide the button so that normal user doesn't mistakenly cancel a long running flink job. This issue adds configuration for hiding the `Cancel Job` button. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-10228) Add metrics for netty direct memory consumption
Ted Yu created FLINK-10228: -- Summary: Add metrics for netty direct memory consumption Key: FLINK-10228 URL: https://issues.apache.org/jira/browse/FLINK-10228 Project: Flink Issue Type: Improvement Reporter: Ted Yu netty direct memory usage can be exposed via metrics so that operator can keep track of memory consumption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10388) RestClientTest sometimes fails with AssertionError
Ted Yu created FLINK-10388:
--
Summary: RestClientTest sometimes fails with AssertionError
Key: FLINK-10388
URL: https://issues.apache.org/jira/browse/FLINK-10388
Project: Flink
Issue Type: Test
Reporter: Ted Yu
Running the test on Linux I got:
{code}
testConnectionTimeout(org.apache.flink.runtime.rest.RestClientTest) Time
elapsed: 1.918 sec <<< FAILURE!
java.lang.AssertionError:
Expected: an instance of
org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException
but:
is a
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedSocketException
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at
org.apache.flink.runtime.rest.RestClientTest.testConnectionTimeout(RestClientTest.java:69)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10389) TaskManagerServicesConfiguration ctor contains self assignment
Ted Yu created FLINK-10389:
--
Summary: TaskManagerServicesConfiguration ctor contains self
assignment
Key: FLINK-10389
URL: https://issues.apache.org/jira/browse/FLINK-10389
Project: Flink
Issue Type: Task
Reporter: Ted Yu
TaskManagerServicesConfiguration has:
{code}
this.systemResourceMetricsEnabled = systemResourceMetricsEnabled;
{code}
There is no systemResourceMetricsEnabled parameter to the ctor.
This was reported by findbugs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10391) MillisOfDay is used in place of instant for LocalTime ctor in AvroKryoSerializerUtils
Ted Yu created FLINK-10391:
--
Summary: MillisOfDay is used in place of instant for LocalTime
ctor in AvroKryoSerializerUtils
Key: FLINK-10391
URL: https://issues.apache.org/jira/browse/FLINK-10391
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
>From the JodaLocalTimeSerializer#write, we serialize getMillisOfDay() value
>from LocalTime.
For read method:
{code}
final int time = input.readInt(true);
return new LocalTime(time,
ISOChronology.getInstanceUTC().withZone(DateTimeZone.UTC));
{code}
It seems
http://joda-time.sourceforge.net/apidocs/org/joda/time/LocalTime.html#fromMillisOfDay(long,%20org.joda.time.Chronology)
should be used instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10446) Use the "guava beta checker" plugin to keep off of @Beta API
Ted Yu created FLINK-10446: -- Summary: Use the "guava beta checker" plugin to keep off of @Beta API Key: FLINK-10446 URL: https://issues.apache.org/jira/browse/FLINK-10446 Project: Flink Issue Type: Task Reporter: Ted Yu The Guava people publish an Error Prone plugin to detect when stuff that's annotated with @Beta gets used. Those things shouldn't be used because the project gives no promises about deprecating before removal. plugin: https://github.com/google/guava-beta-checker -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10467) Upgrade commons-compress to 1.18
Ted Yu created FLINK-10467: -- Summary: Upgrade commons-compress to 1.18 Key: FLINK-10467 URL: https://issues.apache.org/jira/browse/FLINK-10467 Project: Flink Issue Type: Task Reporter: Ted Yu org.apache.commons:commons-compress defines an API for working with compression and archive formats. Affected versions of this package are vulnerable to Directory Traversal. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10468) Potential missing break for PARTITION_CUSTOM in OutputEmitter ctor
Ted Yu created FLINK-10468:
--
Summary: Potential missing break for PARTITION_CUSTOM in
OutputEmitter ctor
Key: FLINK-10468
URL: https://issues.apache.org/jira/browse/FLINK-10468
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
switch (strategy) {
case PARTITION_CUSTOM:
extractedKeys = new Object[1];
case FORWARD:
{code}
It seems a 'break' is missing prior to FORWARD case.
{code}
if (strategy == ShipStrategyType.PARTITION_CUSTOM && partitioner == null) {
throw new NullPointerException("Partitioner must not be null when the
ship strategy is set to custom partitioning.");
}
{code}
Since the above check is for PARTITION_CUSTOM, it seems we can place the check
in the switch statement.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-2912) Potential resource leak due to unclosed RandomAccessFile in StaticFileServerHandler
Ted Yu created FLINK-2912:
-
Summary: Potential resource leak due to unclosed RandomAccessFile
in StaticFileServerHandler
Key: FLINK-2912
URL: https://issues.apache.org/jira/browse/FLINK-2912
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Here is related code in respondAsLeader():
{code}
final RandomAccessFile raf;
try {
raf = new RandomAccessFile(file, "r");
...
long fileLength = raf.length();
...
// write the content.
ctx.write(new DefaultFileRegion(raf.getChannel(), 0, fileLength),
ctx.newProgressivePromise());
{code}
There is no close() call to raf before returning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-2913) Close of ObjectOutputStream should be enclosed in finally block in FsStateBackend
Ted Yu created FLINK-2913:
-
Summary: Close of ObjectOutputStream should be enclosed in finally
block in FsStateBackend
Key: FLINK-2913
URL: https://issues.apache.org/jira/browse/FLINK-2913
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
ObjectOutputStream os = new ObjectOutputStream(outStream);
os.writeObject(state);
os.close();
{code}
If IOException is thrown out of writeObject(), the close() call would be
skipped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-2914) Missing break in ZooKeeperSubmittedJobGraphStore#SubmittedJobGraphsPathCacheListener#childEvent()
Ted Yu created FLINK-2914:
-
Summary: Missing break in
ZooKeeperSubmittedJobGraphStore#SubmittedJobGraphsPathCacheListener#childEvent()
Key: FLINK-2914
URL: https://issues.apache.org/jira/browse/FLINK-2914
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
case CONNECTION_SUSPENDED:
LOG.warn("ZooKeeper connection SUSPENDED. Changes to the submitted
job " +
"graphs are not monitored (temporarily).");
case CONNECTION_LOST:
LOG.warn("ZooKeeper connection LOST. Changes to the submitted job " +
"graphs are not monitored (permanently).");
break;
case CONNECTION_RECONNECTED:
LOG.info("ZooKeeper connection RECONNECTED. Changes to the submitted
job " +
"graphs are monitored again.");
case INITIALIZED:
LOG.info("SubmittedJobGraphsPathCacheListener initialized");
break;
{code}
For CONNECTION_SUSPENDED and CONNECTION_RECONNECTED, the break statement is
missing.
This would result in unrelated event logging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-2963) Dependence on SerializationUtils#deserialize() should be avoided
Ted Yu created FLINK-2963:
-
Summary: Dependence on SerializationUtils#deserialize() should be
avoided
Key: FLINK-2963
URL: https://issues.apache.org/jira/browse/FLINK-2963
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
There is a problem with `SerializationUtils` from Apache Commons
Lang. Here is an open issue where the class will throw a
`ClassNotFoundException` even if the class is in the classpath in a
multiple-classloader environment:
https://issues.apache.org/jira/browse/LANG-1049
{code}
state = (HashMap)
SerializationUtils.deserialize(bais);
./flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/NonKeyedWindowOperator.java
state = (HashMap)
SerializationUtils.deserialize(bais);
./flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java
return SerializationUtils.deserialize(message);
./flink-streaming-java/src/main/java/org/apache/flink/streaming/util/serialization/JavaDefaultStringSchema.java
T copied = SerializationUtils.deserialize(SerializationUtils
./flink-streaming-java/src/test/java/org/apache/flink/streaming/util/MockOutput.java
{code}
We should move away from SerializationUtils.deserialize()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3005) Commons-collections object deserialization remote command execution vulnerability
Ted Yu created FLINK-3005: - Summary: Commons-collections object deserialization remote command execution vulnerability Key: FLINK-3005 URL: https://issues.apache.org/jira/browse/FLINK-3005 Project: Flink Issue Type: Bug Reporter: Ted Yu http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in-common-this-vulnerability/ TL;DR: If you have commons-collections on your classpath and accept and process Java object serialization data, then you may have an exploitable remote command execution vulnerability. Brief search in code base for ObjectInputStream reveals several places where the vulnerability exists. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3014) Return code from InputStream#read() should be checked in PythonStreamer#sendBroadCastVariables
Ted Yu created FLINK-3014:
-
Summary: Return code from InputStream#read() should be checked in
PythonStreamer#sendBroadCastVariables
Key: FLINK-3014
URL: https://issues.apache.org/jira/browse/FLINK-3014
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is related code (there're 3 occurrences in the method):
{code}
in.read(buffer, 0, 4);
checkForError();
{code}
java.io.InputStream.read(byte[], int, int) returns the number of bytes read.
The return value should be checked because checkForError() does this:
{code}
if (getInt(buffer, 0) == -2) {
try { //wait before terminating to ensure that the complete error message
is printed
{code}
Incorrect result may be returned if fewer than 4 bytes of data are read.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3103) Accessing closed in FsStateBackend#FsCheckpointStateOutputStream#flush() should be synchronized
Ted Yu created FLINK-3103:
-
Summary: Accessing closed in
FsStateBackend#FsCheckpointStateOutputStream#flush() should be synchronized
Key: FLINK-3103
URL: https://issues.apache.org/jira/browse/FLINK-3103
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
public void flush() throws IOException {
if (!closed) {
{code}
Lock "FsCheckpointStateOutputStream.this" should be held.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3104) Potential null dereference of outStream in FsStateBackend#CheckpointStateOutputStream#closeAndGetHandle()
Ted Yu created FLINK-3104:
-
Summary: Potential null dereference of outStream in
FsStateBackend#CheckpointStateOutputStream#closeAndGetHandle()
Key: FLINK-3104
URL: https://issues.apache.org/jira/browse/FLINK-3104
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is related code:
{code}
if (outStream == null && pos <= localStateThreshold) {
closed = true;
byte[] bytes = Arrays.copyOf(writeBuffer, pos);
return new ByteStreamStateHandle(bytes);
}
else {
flush();
outStream.close();
{code}
The if condition checks both outStream and pos.
In the else block, outStream may be null, leading to NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3137) Missing break in OutputEmitter ctor
Ted Yu created FLINK-3137:
-
Summary: Missing break in OutputEmitter ctor
Key: FLINK-3137
URL: https://issues.apache.org/jira/browse/FLINK-3137
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
switch (strategy) {
case PARTITION_CUSTOM:
extractedKeys = new Object[1];
case FORWARD:
case PARTITION_HASH:
{code}
Looks like a break is missing following assignment to extractedKeys
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3196) InputStream should be closed in EnvironmentInformation#getRevisionInformation()
Ted Yu created FLINK-3196:
-
Summary: InputStream should be closed in
EnvironmentInformation#getRevisionInformation()
Key: FLINK-3196
URL: https://issues.apache.org/jira/browse/FLINK-3196
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is related code:
{code}
InputStream propFile =
EnvironmentInformation.class.getClassLoader().getResourceAsStream(".version.properties");
if (propFile != null) {
Properties properties = new Properties();
properties.load(propFile);
{code}
propFile should be closed upon leaving the method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3197) InputStream not closed in BinaryInputFormat#createStatistics
Ted Yu created FLINK-3197:
-
Summary: InputStream not closed in
BinaryInputFormat#createStatistics
Key: FLINK-3197
URL: https://issues.apache.org/jira/browse/FLINK-3197
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is related code:
{code}
FSDataInputStream fdis =
file.getPath().getFileSystem().open(file.getPath(), blockInfo.getInfoSize());
fdis.seek(file.getLen() - blockInfo.getInfoSize());
blockInfo.read(new DataInputViewStreamWrapper(fdis));
totalCount += blockInfo.getAccumulatedRecordCount();
{code}
fdis / wrapper should be closed upon leaving the method
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3210) Unnecessary call to deserializer#deserialize() in LegacyFetcher#SimpleConsumerThread#run()
Ted Yu created FLINK-3210:
-
Summary: Unnecessary call to deserializer#deserialize() in
LegacyFetcher#SimpleConsumerThread#run()
Key: FLINK-3210
URL: https://issues.apache.org/jira/browse/FLINK-3210
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is related code:
{code}
byte[] valueBytes;
if (payload == null) {
deletedMessages++;
valueBytes = null;
} else {
...
final T value = deserializer.deserialize(keyBytes, valueBytes,
fp.topic, offset);
{code}
When valueBytes is null, there is no need to call deserializer#deserialize()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3222) Incorrect shift amount in OperatorCheckpointStats#hashCode()
Ted Yu created FLINK-3222:
-
Summary: Incorrect shift amount in
OperatorCheckpointStats#hashCode()
Key: FLINK-3222
URL: https://issues.apache.org/jira/browse/FLINK-3222
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is related code:
{code}
result = 31 * result + (int) (subTaskStats.length ^ (subTaskStats.length
>>> 32));
{code}
subTaskStats.length is an int.
The shift amount is greater than 31 bits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3269) Return value from DataInputStream#read() should be checked in PythonPlanReceiver
Ted Yu created FLINK-3269:
-
Summary: Return value from DataInputStream#read() should be
checked in PythonPlanReceiver
Key: FLINK-3269
URL: https://issues.apache.org/jira/browse/FLINK-3269
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
There're several places where the return value from DataInputStream#read()
isn't checked.
Here is an example from BytesDeserializer :
{code}
byte[] buffer = new byte[size];
input.read(buffer);
return buffer;
{code}
Buffer of size bytes is allocated.
However, the read operation may not fill size bytes.
The return value should be checked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3280) Wrong usage of Boolean.getBoolean()
Ted Yu created FLINK-3280:
-
Summary: Wrong usage of Boolean.getBoolean()
Key: FLINK-3280
URL: https://issues.apache.org/jira/browse/FLINK-3280
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
I observed the problem in both FlinkKafkaProducerBase.java and
FlinkKafkaConsumer09.java
>From FlinkKafkaProducerBase.java :
{code}
if(!Boolean.getBoolean(producerConfig.getProperty(KEY_DISABLE_METRICS,
"false"))) {
{code}
Boolean.getBoolean() would call System.getProperty(). See
http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/lang/Boolean.java#238
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3301) Ineffective synchronization in MessageAcknowledgingSourceBase#restoreState
Ted Yu created FLINK-3301:
-
Summary: Ineffective synchronization in
MessageAcknowledgingSourceBase#restoreState
Key: FLINK-3301
URL: https://issues.apache.org/jira/browse/FLINK-3301
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
public void restoreState(SerializedCheckpointData[] state) throws Exception {
synchronized (pendingCheckpoints) {
pendingCheckpoints = SerializedCheckpointData.toDeque(state,
idSerializer);
{code}
When lock is held on current pendingCheckpoints field, new reference is
assigned to the same field - making the synchronization ineffective.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3325) Unclosed InputStream in CEPPatternOperator#restoreState()
Ted Yu created FLINK-3325:
-
Summary: Unclosed InputStream in CEPPatternOperator#restoreState()
Key: FLINK-3325
URL: https://issues.apache.org/jira/browse/FLINK-3325
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is related code:
{code}
final InputStream is = stream.getState(getUserCodeClassloader());
final ObjectInputStream ois = new ObjectInputStream(is);
final DataInputViewStreamWrapper div = new DataInputViewStreamWrapper(is);
nfa = (NFA)ois.readObject();
{code}
Neither is nor ois is closed upon return from the method
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3329) Unclosed BackupEngine in AbstractRocksDBState#snapshot()
Ted Yu created FLINK-3329:
-
Summary: Unclosed BackupEngine in AbstractRocksDBState#snapshot()
Key: FLINK-3329
URL: https://issues.apache.org/jira/browse/FLINK-3329
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
BackupEngine backupEngine = BackupEngine.open(Env.getDefault(),
new BackupableDBOptions(localBackupPath.getAbsolutePath()));
backupEngine.createNewBackup(db);
{code}
BackupEngine implements AutoCloseable.
backupEngine should be closed upon return from the method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3362) Potential overflow in NettyBufferPool#getNumberOfAllocatedBytes
Ted Yu created FLINK-3362: - Summary: Potential overflow in NettyBufferPool#getNumberOfAllocatedBytes Key: FLINK-3362 URL: https://issues.apache.org/jira/browse/FLINK-3362 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is related code: long allocatedBytes = numChunks * chunkSize; The evaluation of numChunks * chunkSize is done using 32-bit arithmetic. This may result in overflow since the expected type of expression is long. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3392) Unprotected access to elements in ClosableBlockingQueue#size()
Ted Yu created FLINK-3392:
-
Summary: Unprotected access to elements in
ClosableBlockingQueue#size()
Key: FLINK-3392
URL: https://issues.apache.org/jira/browse/FLINK-3392
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is related code:
{code}
public int size() {
return elements.size();
}
{code}
Access to elements should be protected by lock.lock() / lock.unlock() pair.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3433) Return value from ObjectInputStream#read() should be checked in AvroOutputFormat#readObject
Ted Yu created FLINK-3433:
-
Summary: Return value from ObjectInputStream#read() should be
checked in AvroOutputFormat#readObject
Key: FLINK-3433
URL: https://issues.apache.org/jira/browse/FLINK-3433
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is related code:
{code}
if(length != 0) {
byte[] json = new byte[length];
in.read(json);
Schema schema = new Schema.Parser().parse(new String(json));
{code}
The return value from in.read() should be verified against length.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3434) Return value from flinkYarnClient#deploy() should be checked against null
Ted Yu created FLINK-3434:
-
Summary: Return value from flinkYarnClient#deploy() should be
checked against null
Key: FLINK-3434
URL: https://issues.apache.org/jira/browse/FLINK-3434
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
In CliFrontend#getClient() :
{code}
yarnCluster = flinkYarnClient.deploy();
yarnCluster.connectToCluster();
{code}
Looking at FlinkYarnClientBase#deployInternal(), null return is possible:
{code}
try {
org.apache.flink.core.fs.FileSystem.setDefaultScheme(flinkConfiguration);
} catch (IOException e) {
LOG.error("Error while setting the default " +
"filesystem scheme from configuration.", e);
return null;
}
{code}
We should check the return from deploy() before calling connectToCluster().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3525) Missing call to super#close() in TimestampsAndPeriodicWatermarksOperator#close()
Ted Yu created FLINK-3525:
-
Summary: Missing call to super#close() in
TimestampsAndPeriodicWatermarksOperator#close()
Key: FLINK-3525
URL: https://issues.apache.org/jira/browse/FLINK-3525
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is related code:
{code}
public void close() throws Exception {
// emit a final watermark
{code}
AbstractUdfStreamOperator#close() should be called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3576) Upgrade Snappy Java to 1.1.2.1
Ted Yu created FLINK-3576: - Summary: Upgrade Snappy Java to 1.1.2.1 Key: FLINK-3576 URL: https://issues.apache.org/jira/browse/FLINK-3576 Project: Flink Issue Type: Improvement Reporter: Ted Yu Priority: Minor There was a JVM memory leaky problem reported in https://github.com/xerial/snappy-java/issues/131 The above issue has been resolved. 1.1.2.1 was released on Jan 22nd. We should upgrade to this release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3635) Potential null deference in TwitterExample#SelectEnglishAndTokenizeFlatMap#flatMap
Ted Yu created FLINK-3635:
-
Summary: Potential null deference in
TwitterExample#SelectEnglishAndTokenizeFlatMap#flatMap
Key: FLINK-3635
URL: https://issues.apache.org/jira/browse/FLINK-3635
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
JsonNode jsonNode = jsonParser.readValue(value, JsonNode.class);
if (jsonNode.has("user") &&
jsonNode.get("user").get("lang").asText().equals("en")) {
{code}
jsonNode.get("user").get("lang") may return null, leading to
NullPointerException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3668) Potential null deference in HadoopInputFormatBase#createInputSplits()
Ted Yu created FLINK-3668:
-
Summary: Potential null deference in
HadoopInputFormatBase#createInputSplits()
Key: FLINK-3668
URL: https://issues.apache.org/jira/browse/FLINK-3668
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is the code:
{code}
List splits;
try {
splits = this.mapreduceInputFormat.getSplits(jobContext);
} catch (InterruptedException e) {
throw new IOException("Could not get Splits.", e);
}
HadoopInputSplit[] hadoopInputSplits = new HadoopInputSplit[splits.size()];
{code}
If InterruptedException is caught, splits would be null.
Yet, the next line accesses splits.size() without null check.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3732) Potential null deference in ExecutionConfig#equals()
Ted Yu created FLINK-3732:
-
Summary: Potential null deference in ExecutionConfig#equals()
Key: FLINK-3732
URL: https://issues.apache.org/jira/browse/FLINK-3732
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
((restartStrategyConfiguration == null &&
other.restartStrategyConfiguration == null) ||
restartStrategyConfiguration.equals(other.restartStrategyConfiguration)) &&
{code}
If restartStrategyConfiguration is null but other.restartStrategyConfiguration
is not null, the above would result in NPE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3733) registeredTypesWithKryoSerializers is not assigned in ExecutionConfig#deserializeUserCode()
Ted Yu created FLINK-3733:
-
Summary: registeredTypesWithKryoSerializers is not assigned in
ExecutionConfig#deserializeUserCode()
Key: FLINK-3733
URL: https://issues.apache.org/jira/browse/FLINK-3733
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
if (serializedRegisteredTypesWithKryoSerializers != null) {
registeredTypesWithKryoSerializers =
serializedRegisteredTypesWithKryoSerializers.deserializeValue(userCodeClassLoader);
} else {
registeredTypesWithKryoSerializerClasses = new LinkedHashMap<>();
}
{code}
When serializedRegisteredTypesWithKryoSerializers is null,
registeredTypesWithKryoSerializers is not assigned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3734) Unclosed DataInputView in AbstractAlignedProcessingTimeWindowOperator#restoreState()
Ted Yu created FLINK-3734:
-
Summary: Unclosed DataInputView in
AbstractAlignedProcessingTimeWindowOperator#restoreState()
Key: FLINK-3734
URL: https://issues.apache.org/jira/browse/FLINK-3734
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
DataInputView in = inputState.getState(getUserCodeClassloader());
final long nextEvaluationTime = in.readLong();
final long nextSlideTime = in.readLong();
AbstractKeyedTimePanes panes =
createPanes(keySelector, function);
panes.readFromInput(in, keySerializer, stateTypeSerializer);
restoredState = new RestoredState<>(panes, nextEvaluationTime,
nextSlideTime);
}
{code}
DataInputView in is not closed upon return.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3753) KillerWatchDog should not use kill on toKill thread
Ted Yu created FLINK-3753:
-
Summary: KillerWatchDog should not use kill on toKill thread
Key: FLINK-3753
URL: https://issues.apache.org/jira/browse/FLINK-3753
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
// this is harsh, but this watchdog is a last resort
if (toKill.isAlive()) {
toKill.stop();
}
{code}
stop() is deprecated.
See:
https://www.securecoding.cert.org/confluence/display/java/THI05-J.+Do+not+use+Thread.stop()+to+terminate+threads
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3773) Scanners are left unclosed in SqlExplainTest
Ted Yu created FLINK-3773:
-
Summary: Scanners are left unclosed in SqlExplainTest
Key: FLINK-3773
URL: https://issues.apache.org/jira/browse/FLINK-3773
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
e.g.
{code}
String source = new Scanner(new File(testFilePath +
"../../src/test/scala/resources/testFilter0.out"))
{code}
Scanner implements AutoCloseable.
Using try-with-resources would be a good pattern for closing the Scanners.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3781) BlobClient may be left unclosed in BlobCache#deleteGlobal()
Ted Yu created FLINK-3781:
-
Summary: BlobClient may be left unclosed in
BlobCache#deleteGlobal()
Key: FLINK-3781
URL: https://issues.apache.org/jira/browse/FLINK-3781
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
public void deleteGlobal(BlobKey key) throws IOException {
delete(key);
BlobClient bc = createClient();
bc.delete(key);
bc.close();
{code}
If delete() throws IOException, BlobClient would be left inclosed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3801) Upgrade Joda-Time library to 2.9.3
Ted Yu created FLINK-3801: - Summary: Upgrade Joda-Time library to 2.9.3 Key: FLINK-3801 URL: https://issues.apache.org/jira/browse/FLINK-3801 Project: Flink Issue Type: Improvement Reporter: Ted Yu Priority: Minor Currently yoda-time 2.5 is used which was very old. We should upgrade to 2.9.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3809) Missing break in ZooKeeperLeaderRetrievalService#handleStateChange()
Ted Yu created FLINK-3809:
-
Summary: Missing break in
ZooKeeperLeaderRetrievalService#handleStateChange()
Key: FLINK-3809
URL: https://issues.apache.org/jira/browse/FLINK-3809
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
{code}
protected void handleStateChange(ConnectionState newState) {
switch (newState) {
case CONNECTED:
LOG.debug("Connected to ZooKeeper quorum. Leader retrieval can start.");
case SUSPENDED:
LOG.warn("Connection to ZooKeeper suspended. Can no longer retrieve the
leader from " +
"ZooKeeper.");
case RECONNECTED:
LOG.info("Connection to ZooKeeper was reconnected. Leader retrieval can
be restarted.");
case LOST:
LOG.warn("Connection to ZooKeeper lost. Can no longer retrieve the
leader from " +
"ZooKeeper.");
}
{code}
Except for LOST state, the other states would lead to multiple logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3810) Missing break in ZooKeeperLeaderElectionService#handleStateChange()
Ted Yu created FLINK-3810:
-
Summary: Missing break in
ZooKeeperLeaderElectionService#handleStateChange()
Key: FLINK-3810
URL: https://issues.apache.org/jira/browse/FLINK-3810
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
{code}
protected void handleStateChange(ConnectionState newState) {
switch (newState) {
case CONNECTED:
LOG.debug("Connected to ZooKeeper quorum. Leader election can start.");
case SUSPENDED:
LOG.warn("Connection to ZooKeeper suspended. The contender " +
leaderContender.getAddress()
+ "no longer participates in the leader election.");
case RECONNECTED:
LOG.info("Connection to ZooKeeper was reconnected. Leader election can
be restarted.");
case LOST:
// Maybe we have to throw an exception here to terminate the JobManager
LOG.warn("Connection to ZooKeeper lost. The contender " +
leaderContender.getAddress()
+ "no longer participates in the leader election.");
{code}
Any of the first 3 states would result in multiple log output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3928) Potential overflow due to 32-bit int arithmetic
Ted Yu created FLINK-3928:
-
Summary: Potential overflow due to 32-bit int arithmetic
Key: FLINK-3928
URL: https://issues.apache.org/jira/browse/FLINK-3928
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
The following pattern occurs in both LocalClusteringCoefficient.java and
TriangleListing.java :
{code}
int scale = parameters.getInt("scale", DEFAULT_SCALE);
...
long vertexCount = 1 << scale;
{code}
"1 << scale" may overflow with type "int" due to the use of 32-bit arithmetic
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3958) Access to MetricRegistry doesn't have proper synchronization in some classes
Ted Yu created FLINK-3958:
-
Summary: Access to MetricRegistry doesn't have proper
synchronization in some classes
Key: FLINK-3958
URL: https://issues.apache.org/jira/browse/FLINK-3958
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
In GraphiteReporter#getReporter():
{code}
com.codahale.metrics.graphite.GraphiteReporter.Builder builder =
com.codahale.metrics.graphite.GraphiteReporter.forRegistry(registry);
{code}
Access to registry should be protected by lock on
ScheduledDropwizardReporter.this
Similar issue exists in GangliaReporter#getReporter()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-3998) Access to gauges and counters in StatsDReporter#report() is not properly synchronized
Ted Yu created FLINK-3998:
-
Summary: Access to gauges and counters in StatsDReporter#report()
is not properly synchronized
Key: FLINK-3998
URL: https://issues.apache.org/jira/browse/FLINK-3998
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
{code}
for (Map.Entry, String> entry : gauges.entrySet()) {
reportGauge(entry.getValue(), entry.getKey());
}
for (Map.Entry entry : counters.entrySet()) {
reportCounter(entry.getValue(), entry.getKey());
{code}
Access to gauges and counters should be protected by lock on
AbstractReporter.this
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4053) Return value from Connection should be checked against null
Ted Yu created FLINK-4053:
-
Summary: Return value from Connection should be checked against
null
Key: FLINK-4053
URL: https://issues.apache.org/jira/browse/FLINK-4053
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
In RMQSource.java and RMQSink.java, there is code in the following pattern:
{code}
connection = factory.newConnection();
channel = connection.createChannel();
{code}
According to
https://www.rabbitmq.com/releases/rabbitmq-java-client/current-javadoc/com/rabbitmq/client/Connection.html#createChannel()
:
{code}
Returns:
a new channel descriptor, or null if none is available
{code}
The return value should be checked against null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4076) BoltWrapper#dispose() should call AbstractStreamOperator#dispose()
Ted Yu created FLINK-4076:
-
Summary: BoltWrapper#dispose() should call
AbstractStreamOperator#dispose()
Key: FLINK-4076
URL: https://issues.apache.org/jira/browse/FLINK-4076
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
@Override
public void dispose() {
this.bolt.cleanup();
}
{code}
AbstractStreamOperator#dispose() should be called first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4089) Ineffective null check in YarnClusterClient#getApplicationStatus()
Ted Yu created FLINK-4089:
-
Summary: Ineffective null check in
YarnClusterClient#getApplicationStatus()
Key: FLINK-4089
URL: https://issues.apache.org/jira/browse/FLINK-4089
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
if(pollingRunner == null) {
LOG.warn("YarnClusterClient.getApplicationStatus() has been called on an
uninitialized cluster." +
"The system might be in an erroneous state");
}
ApplicationReport lastReport = pollingRunner.getLastReport();
{code}
If pollingRunner is null, NPE would result from the getLastReport() call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4090) Close of OutputStream should be in finally clause in FlinkYarnSessionCli#writeYarnProperties()
Ted Yu created FLINK-4090:
-
Summary: Close of OutputStream should be in finally clause in
FlinkYarnSessionCli#writeYarnProperties()
Key: FLINK-4090
URL: https://issues.apache.org/jira/browse/FLINK-4090
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
try {
OutputStream out = new FileOutputStream(propertiesFile);
properties.store(out, "Generated YARN properties file");
out.close();
} catch (IOException e) {
{code}
The close of out should be in finally cluase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4096) Nested FileOutputStream is not closed in JarFileCreator
Ted Yu created FLINK-4096:
-
Summary: Nested FileOutputStream is not closed in JarFileCreator
Key: FLINK-4096
URL: https://issues.apache.org/jira/browse/FLINK-4096
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
final JarOutputStream jos = new JarOutputStream(new
FileOutputStream(this.outputFile), new Manifest());
{code}
When jos is closed, the FileOutputStream is not automatically closed.
FileOutputStream should be closed explicitly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4258) Potential null pointer dereference in SavepointCoordinator#onFullyAcknowledgedCheckpoint
Ted Yu created FLINK-4258:
-
Summary: Potential null pointer dereference in
SavepointCoordinator#onFullyAcknowledgedCheckpoint
Key: FLINK-4258
URL: https://issues.apache.org/jira/browse/FLINK-4258
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
{code}
if (promise == null) {
LOG.info("Pending savepoint with ID " + checkpoint.getCheckpointID() + "
has been " +
"removed before receiving acknowledgment.");
}
// Sanity check
if (promise.isCompleted()) {
throw new IllegalStateException("Savepoint promise completed");
{code}
Looks like a return statement is missing in the first if block above.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4259) Unclosed FSDataOutputStream in FileCache#copy()
Ted Yu created FLINK-4259:
-
Summary: Unclosed FSDataOutputStream in FileCache#copy()
Key: FLINK-4259
URL: https://issues.apache.org/jira/browse/FLINK-4259
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
try {
FSDataOutputStream lfsOutput = tFS.create(targetPath, false);
FSDataInputStream fsInput = sFS.open(sourcePath);
IOUtils.copyBytes(fsInput, lfsOutput);
{code}
The FSDataOutputStream lfsOutput should be closed upon exit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4309) Potential null pointer dereference in DelegatingConfiguration#keySet()
Ted Yu created FLINK-4309:
-
Summary: Potential null pointer dereference in
DelegatingConfiguration#keySet()
Key: FLINK-4309
URL: https://issues.apache.org/jira/browse/FLINK-4309
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
final int prefixLen = this.prefix == null ? 0 : this.prefix.length();
for (String key : this.backingConfig.keySet()) {
if (key.startsWith(this.prefix)) {
{code}
If this.prefix == null, we would get NPE in startsWith():
{code}
public boolean startsWith(String prefix, int toffset) {
char ta[] = value;
int to = toffset;
char pa[] = prefix.value;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4436) Unclosed DataOutputBuffer in Utils#setTokensFor()
Ted Yu created FLINK-4436:
-
Summary: Unclosed DataOutputBuffer in Utils#setTokensFor()
Key: FLINK-4436
URL: https://issues.apache.org/jira/browse/FLINK-4436
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
DataOutputBuffer dob = new DataOutputBuffer();
credentials.writeTokenStorageToStream(dob);
{code}
dob should be closed upon returning from the method.
YarnApplicationMasterRunner#createTaskManagerContext() has similar issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4437) Lock evasion around lastTriggeredCheckpoint may lead to lost updates to related fields
Ted Yu created FLINK-4437:
-
Summary: Lock evasion around lastTriggeredCheckpoint may lead to
lost updates to related fields
Key: FLINK-4437
URL: https://issues.apache.org/jira/browse/FLINK-4437
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
In CheckpointCoordinator#triggerCheckpoint():
{code}
// make sure the minimum interval between checkpoints has passed
if (lastTriggeredCheckpoint + minPauseBetweenCheckpoints > timestamp) {
{code}
If two threads evaluate 'lastTriggeredCheckpoint + minPauseBetweenCheckpoints >
timestamp' in close proximity before lastTriggeredCheckpoint is updated, the
two threads may have an inconsistent view of "lastTriggeredCheckpoint" and
updates to fields correlated with "lastTriggeredCheckpoint" may be lost.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4482) numUnsuccessfulCheckpointsTriggers is accessed without holding triggerLock
Ted Yu created FLINK-4482:
-
Summary: numUnsuccessfulCheckpointsTriggers is accessed without
holding triggerLock
Key: FLINK-4482
URL: https://issues.apache.org/jira/browse/FLINK-4482
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
In CheckpointCoordinator#stopCheckpointScheduler() :
{code}
synchronized (lock) {
...
numUnsuccessfulCheckpointsTriggers = 0;
{code}
triggerLock is not held in the above operation.
See comment for triggerLock earlier in triggerCheckpoint():
{code}
// we lock with a special lock to make sure that trigger requests do not
overtake each other.
// this is not done with the coordinator-wide lock, because the
'checkpointIdCounter'
// may issue blocking operations. Using a different lock than teh
coordinator-wide lock,
// we avoid blocking the processing of 'acknowledge/decline' messages
during that time.
synchronized (triggerLock) {
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4499) Introduce findbugs maven plugin
Ted Yu created FLINK-4499: - Summary: Introduce findbugs maven plugin Key: FLINK-4499 URL: https://issues.apache.org/jira/browse/FLINK-4499 Project: Flink Issue Type: Improvement Reporter: Ted Yu As suggested by Stephan in FLINK-4482, this issue is to add findbugs-maven-plugin into the build process so that we can detect lack of proper locking and other defects automatically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4531) Thread.stop() is deprecated
Ted Yu created FLINK-4531:
-
Summary: Thread.stop() is deprecated
Key: FLINK-4531
URL: https://issues.apache.org/jira/browse/FLINK-4531
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
In KillerWatchDog#run() :
{code}
if (toKill.isAlive()) {
toKill.stop();
}
{code}
Thread.stop() is deprecated.
https://www.securecoding.cert.org/confluence/display/java/THI05-J.+Do+not+use+Thread.stop()+to+terminate+threads
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4533) Unprotected access to meters in StatsDReporter#report()
Ted Yu created FLINK-4533: - Summary: Unprotected access to meters in StatsDReporter#report() Key: FLINK-4533 URL: https://issues.apache.org/jira/browse/FLINK-4533 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Access to meters in AbstractReporter is protected by AbstractReporter.this. Access to meters in StatsDReporter#report() should do the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4534) Lack of synchronization in BucketingSink#restoreState() w.r.t. state.bucketStates
Ted Yu created FLINK-4534:
-
Summary: Lack of synchronization in BucketingSink#restoreState()
w.r.t. state.bucketStates
Key: FLINK-4534
URL: https://issues.apache.org/jira/browse/FLINK-4534
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Iteration over state.bucketStates is protected by synchronization in other
methods, except for the following in restoreState():
{code}
for (BucketState bucketState : state.bucketStates.values()) {
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-6997) SavepointITCase fails in master branch
Ted Yu created FLINK-6997:
-
Summary: SavepointITCase fails in master branch
Key: FLINK-6997
URL: https://issues.apache.org/jira/browse/FLINK-6997
Project: Flink
Issue Type: Bug
Components: Tests
Reporter: Ted Yu
I got the following test failure (with commit
a0b781461bcf8c2f1d00b93464995f03eda589f1)
{code}
testSavepointForJobWithIteration(org.apache.flink.test.checkpointing.SavepointITCase)
Time elapsed: 8.129 sec <<< ERROR!
java.io.IOException: java.lang.Exception: Failed to complete savepoint
at
org.apache.flink.runtime.testingUtils.TestingCluster.triggerSavepoint(TestingCluster.scala:342)
at
org.apache.flink.runtime.testingUtils.TestingCluster.triggerSavepoint(TestingCluster.scala:316)
at
org.apache.flink.test.checkpointing.SavepointITCase.testSavepointForJobWithIteration(SavepointITCase.java:827)
Caused by: java.lang.Exception: Failed to complete savepoint
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anon$7.apply(JobManager.scala:821)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anon$7.apply(JobManager.scala:805)
at
org.apache.flink.runtime.concurrent.impl.FlinkFuture$5.onComplete(FlinkFuture.java:272)
at akka.dispatch.OnComplete.internal(Future.scala:247)
at akka.dispatch.OnComplete.internal(Future.scala:245)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.Exception: Failed to trigger savepoint: Not all required
tasks are currently running.
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.triggerSavepoint(CheckpointCoordinator.java:382)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:800)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at
org.apache.flink.runtime.testingUtils.TestingJobManagerLike$$anonfun$handleTestingMessage$1.applyOrElse(TestingJobManagerLike.scala:95)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162)
at
org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:38)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7049) TestingApplicationMaster keeps running after integration tests finish
Ted Yu created FLINK-7049:
-
Summary: TestingApplicationMaster keeps running after integration
tests finish
Key: FLINK-7049
URL: https://issues.apache.org/jira/browse/FLINK-7049
Project: Flink
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
After integration tests finish, TestingApplicationMaster is still running.
Toward the end of
flink-yarn-tests/target/flink-yarn-tests-ha/flink-yarn-tests-ha-logDir-nm-1_0/application_1498768839874_0001/container_1498768839874_0001_03_01/jobmanager.log
:
{code}
2017-06-29 22:09:49,681 INFO org.apache.zookeeper.ClientCnxn
- Opening socket connection to server 127.0.0.1/127.0.0.1:46165
2017-06-29 22:09:49,681 ERROR
org.apache.flink.shaded.org.apache.curator.ConnectionState- Authentication
failed
2017-06-29 22:09:49,682 WARN org.apache.zookeeper.ClientCnxn
- Session 0x0 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-06-29 22:09:50,782 WARN org.apache.zookeeper.ClientCnxn
- SASL configuration failed:
javax.security.auth.login.LoginException: No JAAS configuration section named
'Client' was found in specified JAAS configuration file:
'/tmp/jaas-3597644653611245612.conf'. Will continue connection to Zookeeper
server without SASL authentication, if Zookeeper server allows it.
2017-06-29 22:09:50,782 INFO org.apache.zookeeper.ClientCnxn
- Opening socket connection to server 127.0.0.1/127.0.0.1:46165
2017-06-29 22:09:50,782 ERROR
org.apache.flink.shaded.org.apache.curator.ConnectionState- Authentication
failed
2017-06-29 22:09:50,783 WARN org.apache.zookeeper.ClientCnxn
- Session 0x0 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7197) Missing call to GraphAlgorithmWrappingBase#canMergeConfigurationWith()
Ted Yu created FLINK-7197:
-
Summary: Missing call to
GraphAlgorithmWrappingBase#canMergeConfigurationWith()
Key: FLINK-7197
URL: https://issues.apache.org/jira/browse/FLINK-7197
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
In
flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/asm/translate/TranslateVertexValues.java
:
{code}
protected boolean canMergeConfigurationWith(GraphAlgorithmWrappingBase other)
{
super.mergeConfiguration(other);
{code}
GraphAlgorithmWrappingBase#canMergeConfigurationWith() should be called.
There is similar issue in TranslateGraphIds.java
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7260) Match not exhaustive in WindowJoinUtil.scala
Ted Yu created FLINK-7260:
-
Summary: Match not exhaustive in WindowJoinUtil.scala
Key: FLINK-7260
URL: https://issues.apache.org/jira/browse/FLINK-7260
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is related code:
{code}
[WARNING]
/mnt/disk2/a/flink/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/join/WindowJoinUtil.scala:296:
warning: match may not be exhaustive.
[WARNING] It would fail on the following inputs: BETWEEN, CREATE_VIEW, DEFAULT,
DESCRIBE_SCHEMA, DOT, EXCEPT, EXTRACT, GREATEST, LAST_VALUE,
MAP_QUERY_CONSTRUCTOR, NOT, NOT_EQUALS, NULLS_LAST, PATTERN_ALTER, PREV,
SIMILAR, SUM0, TIMESTAMP_DIFF, UNION, VAR_SAMP
[WARNING] timePred.pred.getKind match {
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7402) Ineffective null check in NettyMessage#write()
Ted Yu created FLINK-7402:
-
Summary: Ineffective null check in NettyMessage#write()
Key: FLINK-7402
URL: https://issues.apache.org/jira/browse/FLINK-7402
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is the null check in finally block:
{code}
finally {
if (buffer != null) {
buffer.recycle();
}
{code}
But buffer has been dereferenced in the try block without guard.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7432) Unclosed HighAvailabilityServices instance in QueryableStateClient
Ted Yu created FLINK-7432:
-
Summary: Unclosed HighAvailabilityServices instance in
QueryableStateClient
Key: FLINK-7432
URL: https://issues.apache.org/jira/browse/FLINK-7432
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
public QueryableStateClient(Configuration config) throws Exception {
this(config, HighAvailabilityServicesUtils.createHighAvailabilityServices(
config, Executors.directExecutor(),
HighAvailabilityServicesUtils.AddressResolution.TRY_ADDRESS_RESOLUTION));
{code}
The HighAvailabilityServices instance is only used for calling
getJobManagerLeaderRetriever().
The instance should be closed upon leaving QueryableStateClient ctor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7433) Lack of synchronization accessing inProgress in JobCancellationWithSavepointHandlers
Ted Yu created FLINK-7433:
-
Summary: Lack of synchronization accessing inProgress in
JobCancellationWithSavepointHandlers
Key: FLINK-7433
URL: https://issues.apache.org/jira/browse/FLINK-7433
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
try {
if (throwable != null) {
completed.put(requestId, throwable);
} else {
completed.put(requestId, path);
}
} finally {
inProgress.remove(jobId);
}
{code}
The call to inProgress.remove(jobId) should be protected by lock object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7437) Upgrade calcite to calcite 1.14
Ted Yu created FLINK-7437: - Summary: Upgrade calcite to calcite 1.14 Key: FLINK-7437 URL: https://issues.apache.org/jira/browse/FLINK-7437 Project: Flink Issue Type: Task Reporter: Ted Yu The upcoming calcite 1.14 contains important fixes. e.g. CALCITE-1859 would allow HBaseConnectorITCase to pass. This issue is to upgrade Calcite to 1.14 release -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7438) Some classes are eclipsed by classes in package scala
Ted Yu created FLINK-7438:
-
Summary: Some classes are eclipsed by classes in package scala
Key: FLINK-7438
URL: https://issues.apache.org/jira/browse/FLINK-7438
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Noticed the following during compilation:
{code}
[WARNING]
/flink/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/WindowedStream.scala:33:
warning: imported `OutputTag' is permanently hidden by definition of
object OutputTag in package scala
[WARNING] import org.apache.flink.util.{Collector, OutputTag}
[WARNING] ^
[WARNING]
/flink/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/WindowedStream.scala:33:
warning: imported `OutputTag' is permanently hidden by definition of
class OutputTag in package scala
[WARNING] import org.apache.flink.util.{Collector, OutputTag}
{code}
We should avoid the warning e.r.t. OutputTag.
There may be other occurrences of similar warning.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7488) TaskManagerHeapSizeCalculationJavaBashTest sometimes fails
Ted Yu created FLINK-7488:
-
Summary: TaskManagerHeapSizeCalculationJavaBashTest sometimes fails
Key: FLINK-7488
URL: https://issues.apache.org/jira/browse/FLINK-7488
Project: Flink
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
{code}
TaskManagerHeapSizeCalculationJavaBashTest.compareHeapSizeShellScriptWithJava:110->compareHeapSizeJavaVsScript:275
Different heap sizes with configuration:
{taskmanager.network.memory.fraction=0.1, taskmanager.memory.off-heap=false,
taskmanager.memory.fraction=0.7, taskmanager.memory.size=-1,
taskmanager.network.memory.max=1073741824, taskmanager.heap.mb=1000,
taskmanager.network.memory.min=67108864} expected:<[]1000> but was:<[Setting
HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set.Using the
result of 'hadoop classpath' to augment the Hadoop classpath:
/usr/hdp/2.5.0.0-1245/hadoop/conf:/usr/hdp/2.5.0.0-1245/hadoop/lib/*:/usr/hdp/2.5.0.0-1245/hadoop/.//*:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/./:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/.//*:/usr/hdp/2.5.0.0-1245/hadoop-yarn/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-yarn/.//*:/usr/hdp/2.5.0.0-1245/hadoop-mapreduce/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-mapreduce/.//*:/usr/hdp/2.5.0.0-1245/tez/*:/usr/hdp/2.5.0.0-1245/tez/lib/*:/usr/hdp/2.5.0.0-1245/tez/conf]1000>
TaskManagerHeapSizeCalculationJavaBashTest.compareNetworkBufShellScriptWithJava:81->compareNetworkBufJavaVsScript:235
Different network buffer memory sizes with configuration:
{taskmanager.network.memory.fraction=0.1, taskmanager.memory.off-heap=false,
taskmanager.memory.fraction=0.7, taskmanager.memory.size=-1,
taskmanager.network.memory.max=1073741824, taskmanager.heap.mb=1000,
taskmanager.network.memory.min=67108864} expected:<[]104857600> but
was:<[Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was
set.Using the result of 'hadoop classpath' to augment the Hadoop classpath:
/usr/hdp/2.5.0.0-1245/hadoop/conf:/usr/hdp/2.5.0.0-1245/hadoop/lib/*:/usr/hdp/2.5.0.0-1245/hadoop/.//*:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/./:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/.//*:/usr/hdp/2.5.0.0-1245/hadoop-yarn/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-yarn/.//*:/usr/hdp/2.5.0.0-1245/hadoop-mapreduce/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-mapreduce/.//*:/usr/hdp/2.5.0.0-1245/tez/*:/usr/hdp/2.5.0.0-1245/tez/lib/*:/usr/hdp/2.5.0.0-1245/tez/conf]104857600>
{code}
$HADOOP_CONF_DIR was not set prior to running the test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7495) AbstractUdfStreamOperator#initializeState() should be called in AsyncWaitOperator#initializeState()
Ted Yu created FLINK-7495:
-
Summary: AbstractUdfStreamOperator#initializeState() should be
called in AsyncWaitOperator#initializeState()
Key: FLINK-7495
URL: https://issues.apache.org/jira/browse/FLINK-7495
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
recoveredStreamElements = context
.getOperatorStateStore()
.getListState(new ListStateDescriptor<>(STATE_NAME,
inStreamElementSerializer));
{code}
Call to AbstractUdfStreamOperator#initializeState() should be added in the
beginning
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7525) Add config option to disable Cancel functionality on UI
Ted Yu created FLINK-7525: - Summary: Add config option to disable Cancel functionality on UI Key: FLINK-7525 URL: https://issues.apache.org/jira/browse/FLINK-7525 Project: Flink Issue Type: Improvement Reporter: Ted Yu In this email thread http://search-hadoop.com/m/Flink/VkLeQlf0QOnc7YA?subj=Security+Control+of+running+Flink+Jobs+on+Flink+UI , Raja was asking for a way to control the way users cancel Job(s). Robert proposed adding a config option which disables the Cancel functionality. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7588) Document RocksDB tuning for spinning disks
Ted Yu created FLINK-7588: - Summary: Document RocksDB tuning for spinning disks Key: FLINK-7588 URL: https://issues.apache.org/jira/browse/FLINK-7588 Project: Flink Issue Type: Improvement Reporter: Ted Yu In docs/ops/state/large_state_tuning.md , it was mentioned that: bq. the default configuration is tailored towards SSDs and performs suboptimal on spinning disks We should add recommendation targeting spinning disks: https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#difference-of-spinning-disk -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7631) Ineffective check in PageRank#open()
Ted Yu created FLINK-7631:
-
Summary: Ineffective check in PageRank#open()
Key: FLINK-7631
URL: https://issues.apache.org/jira/browse/FLINK-7631
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
>From
>flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/library/linkanalysis/PageRank.java
> :
{code}
this.vertexCount = vertexCountIterator.hasNext() ?
vertexCountIterator.next().getValue() : 0;
this.uniformlyDistributedScore = ((1 - dampingFactor) + dampingFactor *
sumOfSinks) / this.vertexCount;
{code}
The check for vertexCountIterator.hasNext() should enclose the assignments to
both this.vertexCount and this.uniformlyDistributedScore
Otherwise there may be divide by zero error.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7642) Upgrade maven surefire plugin to 2.19.1
Ted Yu created FLINK-7642: - Summary: Upgrade maven surefire plugin to 2.19.1 Key: FLINK-7642 URL: https://issues.apache.org/jira/browse/FLINK-7642 Project: Flink Issue Type: Improvement Reporter: Ted Yu Surefire 2.19 release introduced more useful test filters which would let us run a subset of the test. This issue is for upgrading maven surefire plugin to 2.19.1 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7659) Unprotected access to inProgress in JobCancellationWithSavepointHandlers#handleNewRequest
Ted Yu created FLINK-7659:
-
Summary: Unprotected access to inProgress in
JobCancellationWithSavepointHandlers#handleNewRequest
Key: FLINK-7659
URL: https://issues.apache.org/jira/browse/FLINK-7659
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
} finally {
inProgress.remove(jobId);
}
{code}
A little lower, in another finally block, there is:
{code}
synchronized (lock) {
if (!success) {
inProgress.remove(jobId);
{code}
which is correct.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7679) Upgrade maven enforcer plugin to 3.0.0-M1
Ted Yu created FLINK-7679:
-
Summary: Upgrade maven enforcer plugin to 3.0.0-M1
Key: FLINK-7679
URL: https://issues.apache.org/jira/browse/FLINK-7679
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
I got the following build error against Java 9:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce (enforce-maven) on
project flink-parent: Execution enforce-maven of goal
org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce failed: An API
incompatibility was encountered while executing
org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce:
java.lang.ExceptionInInitializerError: null
[ERROR] -
[ERROR] realm =plugin>org.apache.maven.plugins:maven-enforcer-plugin:1.4.1
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] =
file:/home/hbase/.m2/repository/org/apache/maven/plugins/maven-enforcer-plugin/1.4.1/maven-enforcer-plugin-1.4.1.jar
{code}
Upgrading maven enforcer plugin to 3.0.0-M1 would get over the above error.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7775) Remove unreferenced method PermanentBlobCache#getNumberOfCachedJobs
Ted Yu created FLINK-7775:
-
Summary: Remove unreferenced method
PermanentBlobCache#getNumberOfCachedJobs
Key: FLINK-7775
URL: https://issues.apache.org/jira/browse/FLINK-7775
Project: Flink
Issue Type: Task
Reporter: Ted Yu
Priority: Minor
{code}
public int getNumberOfCachedJobs() {
return jobRefCounters.size();
}
{code}
The method is not used.
We should remove it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7795) Utilize error-prone to discover common coding mistakes
Ted Yu created FLINK-7795: - Summary: Utilize error-prone to discover common coding mistakes Key: FLINK-7795 URL: https://issues.apache.org/jira/browse/FLINK-7795 Project: Flink Issue Type: Improvement Reporter: Ted Yu http://errorprone.info/ is a tool which detects common coding mistakes. We should incorporate into Flink build. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7796) RocksDBKeyedStateBackend#RocksDBFullSnapshotOperation should close snapshotCloseableRegistry
Ted Yu created FLINK-7796: - Summary: RocksDBKeyedStateBackend#RocksDBFullSnapshotOperation should close snapshotCloseableRegistry Key: FLINK-7796 URL: https://issues.apache.org/jira/browse/FLINK-7796 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor snapshotCloseableRegistry, being CloseableRegistry, depends on invocation of close() method to release certain resource. It seems close() can be called from releaseSnapshotResources() -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7877) Fix compilation against the Hadoop 3 beta1 release
Ted Yu created FLINK-7877:
-
Summary: Fix compilation against the Hadoop 3 beta1 release
Key: FLINK-7877
URL: https://issues.apache.org/jira/browse/FLINK-7877
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
When compiling against hadoop 3.0.0-beta1, I got:
{code}
[ERROR]
/mnt/disk2/a/flink/flink-yarn/src/test/java/org/apache/flink/yarn/UtilsTest.java:[224,16]
org.apache.flink.yarn.UtilsTest.TestingContainer is not abstract and does not
override abstract method
setExecutionType(org.apache.hadoop.yarn.api.records.ExecutionType) in
org.apache.hadoop.yarn.api.records.Container
{code}
There may other hadoop API(s) that need adjustment.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7897) Consider using nio.Files for file deletion in TransientBlobCleanupTask
Ted Yu created FLINK-7897: - Summary: Consider using nio.Files for file deletion in TransientBlobCleanupTask Key: FLINK-7897 URL: https://issues.apache.org/jira/browse/FLINK-7897 Project: Flink Issue Type: Improvement Reporter: Ted Yu Priority: Minor nio.Files#delete() provides better clue as to why the deletion may fail: https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#delete(java.nio.file.Path) Depending on the potential exception, the call to localFile.exists() may be skipped. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7917) The return of taskInformationOrBlobKey should be placed inside synchronized in ExecutionJobVertex
Ted Yu created FLINK-7917:
-
Summary: The return of taskInformationOrBlobKey should be placed
inside synchronized in ExecutionJobVertex
Key: FLINK-7917
URL: https://issues.apache.org/jira/browse/FLINK-7917
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Currently:
{code}
}
return taskInformationOrBlobKey;
{code}
The return should be placed inside synchronized block.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7936) Lack of synchronization w.r.t. taskManagers in MetricStore#add()
Ted Yu created FLINK-7936:
-
Summary: Lack of synchronization w.r.t. taskManagers in
MetricStore#add()
Key: FLINK-7936
URL: https://issues.apache.org/jira/browse/FLINK-7936
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
String tmID = ((QueryScopeInfo.TaskManagerQueryScopeInfo)
info).taskManagerID;
tm = taskManagers.computeIfAbsent(tmID, k -> new
TaskManagerMetricStore());
{code}
In other places, access to taskManagers is protected by lock on MetricStore.this
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-7966) Create flink-connector-kafka-1.0 submodule
Ted Yu created FLINK-7966:
-
Summary: Create flink-connector-kafka-1.0 submodule
Key: FLINK-7966
URL: https://issues.apache.org/jira/browse/FLINK-7966
Project: Flink
Issue Type: Improvement
Reporter: Ted Yu
Priority: Major
Kafka 1.0 has been released.
When compiling flink-connector-kafka-0.11 against 1.0.0 release, I got:
{code}
[ERROR]
/a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[468,30]
cannot find symbol
[ERROR] symbol: method setTransactionTimeout(long)
[ERROR]
/a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[469,30]
cannot find symbol
[ERROR] symbol: method enableTransactionTimeoutWarnings(double)
[ERROR]
/a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[509,9]
method does not override or implement a method from a supertype
[ERROR]
/a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[511,22]
cannot find symbol
[ERROR] symbol: method ignoreFailuresAfterTransactionTimeout()
[ERROR]
/a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[586,66]
cannot find symbol
[ERROR] symbol: method currentTransaction()
[ERROR] location: class
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011
[ERROR]
/a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[619,82]
cannot find symbol
[ERROR] symbol: method currentTransaction()
[ERROR] location: class
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011
[ERROR]
/a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[895,66]
cannot find symbol
[ERROR] symbol: method currentTransaction()
[ERROR] location: class
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011
{code}
We should create flink-connector-kafka-1.0 submodule to accommodate the new
Kafka release
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8036) Consider using gradle to build Flink
Ted Yu created FLINK-8036: - Summary: Consider using gradle to build Flink Key: FLINK-8036 URL: https://issues.apache.org/jira/browse/FLINK-8036 Project: Flink Issue Type: Improvement Reporter: Ted Yu Here is summary from Lukasz over this thread (http://search-hadoop.com/m/Beam/gfKHFVh4NM151XIu1?subj=Re+DISCUSS+Move+away+from+Apache+Maven+as+build+tool) w.r.t. performance boost from using gradle: Maven performs parallelization at the module level, an entire module needs to complete before any dependent modules can start, this means running all the checks like findbugs, checkstyle, tests need to finish. Gradle has task level parallelism between subprojects which means that as soon as the compile and shade steps are done for a project, and dependent subprojects can typically start. This means that we get increased parallelism due to not needing to wait for findbugs, checkstyle, tests to run. I typically see ~20 tasks (at peak) running on my desktop in parallel. Flink should consider using gradle - on Linux with SSD, a clean build takes an hour. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8037) Missing cast in integer arithmetic in TransactionalIdsGenerator#generateIdsToAbort
Ted Yu created FLINK-8037:
-
Summary: Missing cast in integer arithmetic in
TransactionalIdsGenerator#generateIdsToAbort
Key: FLINK-8037
URL: https://issues.apache.org/jira/browse/FLINK-8037
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
public Set generateIdsToAbort() {
Set idsToAbort = new HashSet<>();
for (int i = 0; i < safeScaleDownFactor; i++) {
idsToAbort.addAll(generateIdsToUse(i * poolSize * totalNumberOfSubtasks));
{code}
The operands are integers where generateIdsToUse() expects long parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8048) RAT check complaint
Ted Yu created FLINK-8048:
-
Summary: RAT check complaint
Key: FLINK-8048
URL: https://issues.apache.org/jira/browse/FLINK-8048
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Running {{mvn rat:check}} gives warning about the following files:
test-infra/end-to-end-test/test-data/words
.editorconfig
.github/PULL_REQUEST_TEMPLATE.md
.github/CONTRIBUTING.md
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8075) Redundant clone() in Configuration ctor
Ted Yu created FLINK-8075:
-
Summary: Redundant clone() in Configuration ctor
Key: FLINK-8075
URL: https://issues.apache.org/jira/browse/FLINK-8075
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
In
flink-filesystems/flink-s3-fs-presto/src/main/java/org/apache/hadoop/conf/Configuration.java
, at line 703:
{code}
public Configuration(Configuration other) {
this.resources = (ArrayList) other.resources.clone();
synchronized(other) {
if (other.properties != null) {
this.properties = (Properties)other.properties.clone();
}
{code}
The first clone() call is without synchronization and without null check.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8100) Consider introducing log4j-extras
Ted Yu created FLINK-8100: - Summary: Consider introducing log4j-extras Key: FLINK-8100 URL: https://issues.apache.org/jira/browse/FLINK-8100 Project: Flink Issue Type: Improvement Reporter: Ted Yu log4j-extras allows log rotation as well as compression. https://logging.apache.org/log4j/extras/download.html We should consider using log4j-extras. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8146) Potential resource leak in PythonPlanBinder#unzipPythonLibrary
Ted Yu created FLINK-8146:
-
Summary: Potential resource leak in
PythonPlanBinder#unzipPythonLibrary
Key: FLINK-8146
URL: https://issues.apache.org/jira/browse/FLINK-8146
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
{code}
while (entry != null) {
...
}
zis.closeEntry();
{code}
Looking at the catch block inside the loop, it seems the intention is to close
zis upon getting exception.
zis.close() should be called outside the loop.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8198) Useless check against -1 in LimitedConnectionsFileSystem#fromConfig
Ted Yu created FLINK-8198:
-
Summary: Useless check against -1 in
LimitedConnectionsFileSystem#fromConfig
Key: FLINK-8198
URL: https://issues.apache.org/jira/browse/FLINK-8198
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
return new ConnectionLimitingSettings(
totalLimit == -1 ? 0 : totalLimit,
limitIn == -1 ? 0 : limitIn,
limitOut == -1 ? 0 : limitOut,
openTimeout,
{code}
If any of the 3 variables is negative, control would have returned in the if
block above.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8335) Upgrade hbase connector dependency to 1.4.0
Ted Yu created FLINK-8335: - Summary: Upgrade hbase connector dependency to 1.4.0 Key: FLINK-8335 URL: https://issues.apache.org/jira/browse/FLINK-8335 Project: Flink Issue Type: Improvement Reporter: Ted Yu Priority: Minor hbase 1.4.0 has been released. 1.4.0 shows speed improvement over previous 1.x releases. This issue is to upgrade the dependency to 1.4.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8394) Lack of synchronization accessing expectedRecord in ReceiverThread#shutdown
Ted Yu created FLINK-8394:
-
Summary: Lack of synchronization accessing expectedRecord in
ReceiverThread#shutdown
Key: FLINK-8394
URL: https://issues.apache.org/jira/browse/FLINK-8394
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
public void shutdown() {
running = false;
interrupt();
expectedRecord.complete(0L);
{code}
Access to expectedRecord should be protected by synchronization, as done on
other methods.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8415) Unprotected access to recordsToSend in LongRecordWriterThread#shutdown()
Ted Yu created FLINK-8415:
-
Summary: Unprotected access to recordsToSend in
LongRecordWriterThread#shutdown()
Key: FLINK-8415
URL: https://issues.apache.org/jira/browse/FLINK-8415
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
public void shutdown() {
running = false;
recordsToSend.complete(0L);
{code}
In other methods, access to recordsToSend is protected by synchronized keyword.
shutdown() should do the same.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8464) TestRaftReconfigurationWithSimulatedRpc fails with assertion error
Ted Yu created FLINK-8464:
-
Summary: TestRaftReconfigurationWithSimulatedRpc fails with
assertion error
Key: FLINK-8464
URL: https://issues.apache.org/jira/browse/FLINK-8464
Project: Flink
Issue Type: Test
Reporter: Ted Yu
Attachments: ratis-8464.tst
As of commit 7b3a9a6f5f8e8075727d84e3ddeae7b594eda89c, I observed the following
:
{code}
testRevertConfigurationChange(org.apache.ratis.server.simulation.TestRaftReconfigurationWithSimulatedRpc)
Time elapsed: 2.119 sec <<< FAILURE!
java.lang.AssertionError: 1 0 expected: but was:
{code}
1 was confIndex and 0 was log.getLastCommittedIndex()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-8554) Upgrade AWS SDK
Ted Yu created FLINK-8554:
-
Summary: Upgrade AWS SDK
Key: FLINK-8554
URL: https://issues.apache.org/jira/browse/FLINK-8554
Project: Flink
Issue Type: Improvement
Reporter: Ted Yu
AWS SDK 1.11.271 fixes a lot of bugs.
One of which would exhibit the following:
{code}
Caused by: java.lang.NullPointerException
at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729)
at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67)
at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-8661) Replace Collections.EMPTY_MAP with Collections.emptyMap()
Ted Yu created FLINK-8661: - Summary: Replace Collections.EMPTY_MAP with Collections.emptyMap() Key: FLINK-8661 URL: https://issues.apache.org/jira/browse/FLINK-8661 Project: Flink Issue Type: Bug Reporter: Ted Yu The use of Collections.EMPTY_SET and Collections.EMPTY_MAP often causes unchecked assignment and it should be replaced with Collections.emptySet() and Collections.emptyMap() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8687) MaterializedCollectStreamResult#retrievePage should take resultLock
Ted Yu created FLINK-8687: - Summary: MaterializedCollectStreamResult#retrievePage should take resultLock Key: FLINK-8687 URL: https://issues.apache.org/jira/browse/FLINK-8687 Project: Flink Issue Type: Bug Reporter: Ted Yu Currently MaterializedCollectStreamResult#retrievePage checks page range and calls snapshot.subList() without holding resultLock. resultLock should be taken. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
