[jira] [Created] (FLINK-32025) Make job cancellation button on UI configurable
Ted Yu created FLINK-32025: -- Summary: Make job cancellation button on UI configurable Key: FLINK-32025 URL: https://issues.apache.org/jira/browse/FLINK-32025 Project: Flink Issue Type: Improvement Reporter: Ted Yu On the flink job UI, there is `Cancel Job` button. When the job UI is shown to users, it is desirable to hide the button so that normal user doesn't mistakenly cancel a long running flink job. This issue adds configuration for hiding the `Cancel Job` button. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-10228) Add metrics for netty direct memory consumption
Ted Yu created FLINK-10228: -- Summary: Add metrics for netty direct memory consumption Key: FLINK-10228 URL: https://issues.apache.org/jira/browse/FLINK-10228 Project: Flink Issue Type: Improvement Reporter: Ted Yu netty direct memory usage can be exposed via metrics so that operator can keep track of memory consumption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10388) RestClientTest sometimes fails with AssertionError
Ted Yu created FLINK-10388: -- Summary: RestClientTest sometimes fails with AssertionError Key: FLINK-10388 URL: https://issues.apache.org/jira/browse/FLINK-10388 Project: Flink Issue Type: Test Reporter: Ted Yu Running the test on Linux I got: {code} testConnectionTimeout(org.apache.flink.runtime.rest.RestClientTest) Time elapsed: 1.918 sec <<< FAILURE! java.lang.AssertionError: Expected: an instance of org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException but: is a org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedSocketException at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.junit.Assert.assertThat(Assert.java:956) at org.junit.Assert.assertThat(Assert.java:923) at org.apache.flink.runtime.rest.RestClientTest.testConnectionTimeout(RestClientTest.java:69) {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10389) TaskManagerServicesConfiguration ctor contains self assignment
Ted Yu created FLINK-10389: -- Summary: TaskManagerServicesConfiguration ctor contains self assignment Key: FLINK-10389 URL: https://issues.apache.org/jira/browse/FLINK-10389 Project: Flink Issue Type: Task Reporter: Ted Yu TaskManagerServicesConfiguration has: {code} this.systemResourceMetricsEnabled = systemResourceMetricsEnabled; {code} There is no systemResourceMetricsEnabled parameter to the ctor. This was reported by findbugs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10391) MillisOfDay is used in place of instant for LocalTime ctor in AvroKryoSerializerUtils
Ted Yu created FLINK-10391: -- Summary: MillisOfDay is used in place of instant for LocalTime ctor in AvroKryoSerializerUtils Key: FLINK-10391 URL: https://issues.apache.org/jira/browse/FLINK-10391 Project: Flink Issue Type: Bug Reporter: Ted Yu >From the JodaLocalTimeSerializer#write, we serialize getMillisOfDay() value >from LocalTime. For read method: {code} final int time = input.readInt(true); return new LocalTime(time, ISOChronology.getInstanceUTC().withZone(DateTimeZone.UTC)); {code} It seems http://joda-time.sourceforge.net/apidocs/org/joda/time/LocalTime.html#fromMillisOfDay(long,%20org.joda.time.Chronology) should be used instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10446) Use the "guava beta checker" plugin to keep off of @Beta API
Ted Yu created FLINK-10446: -- Summary: Use the "guava beta checker" plugin to keep off of @Beta API Key: FLINK-10446 URL: https://issues.apache.org/jira/browse/FLINK-10446 Project: Flink Issue Type: Task Reporter: Ted Yu The Guava people publish an Error Prone plugin to detect when stuff that's annotated with @Beta gets used. Those things shouldn't be used because the project gives no promises about deprecating before removal. plugin: https://github.com/google/guava-beta-checker -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10467) Upgrade commons-compress to 1.18
Ted Yu created FLINK-10467: -- Summary: Upgrade commons-compress to 1.18 Key: FLINK-10467 URL: https://issues.apache.org/jira/browse/FLINK-10467 Project: Flink Issue Type: Task Reporter: Ted Yu org.apache.commons:commons-compress defines an API for working with compression and archive formats. Affected versions of this package are vulnerable to Directory Traversal. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10468) Potential missing break for PARTITION_CUSTOM in OutputEmitter ctor
Ted Yu created FLINK-10468: -- Summary: Potential missing break for PARTITION_CUSTOM in OutputEmitter ctor Key: FLINK-10468 URL: https://issues.apache.org/jira/browse/FLINK-10468 Project: Flink Issue Type: Bug Reporter: Ted Yu Here is related code: {code} switch (strategy) { case PARTITION_CUSTOM: extractedKeys = new Object[1]; case FORWARD: {code} It seems a 'break' is missing prior to FORWARD case. {code} if (strategy == ShipStrategyType.PARTITION_CUSTOM && partitioner == null) { throw new NullPointerException("Partitioner must not be null when the ship strategy is set to custom partitioning."); } {code} Since the above check is for PARTITION_CUSTOM, it seems we can place the check in the switch statement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-2912) Potential resource leak due to unclosed RandomAccessFile in StaticFileServerHandler
Ted Yu created FLINK-2912: - Summary: Potential resource leak due to unclosed RandomAccessFile in StaticFileServerHandler Key: FLINK-2912 URL: https://issues.apache.org/jira/browse/FLINK-2912 Project: Flink Issue Type: Bug Reporter: Ted Yu Here is related code in respondAsLeader(): {code} final RandomAccessFile raf; try { raf = new RandomAccessFile(file, "r"); ... long fileLength = raf.length(); ... // write the content. ctx.write(new DefaultFileRegion(raf.getChannel(), 0, fileLength), ctx.newProgressivePromise()); {code} There is no close() call to raf before returning. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-2913) Close of ObjectOutputStream should be enclosed in finally block in FsStateBackend
Ted Yu created FLINK-2913: - Summary: Close of ObjectOutputStream should be enclosed in finally block in FsStateBackend Key: FLINK-2913 URL: https://issues.apache.org/jira/browse/FLINK-2913 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} ObjectOutputStream os = new ObjectOutputStream(outStream); os.writeObject(state); os.close(); {code} If IOException is thrown out of writeObject(), the close() call would be skipped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-2914) Missing break in ZooKeeperSubmittedJobGraphStore#SubmittedJobGraphsPathCacheListener#childEvent()
Ted Yu created FLINK-2914: - Summary: Missing break in ZooKeeperSubmittedJobGraphStore#SubmittedJobGraphsPathCacheListener#childEvent() Key: FLINK-2914 URL: https://issues.apache.org/jira/browse/FLINK-2914 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} case CONNECTION_SUSPENDED: LOG.warn("ZooKeeper connection SUSPENDED. Changes to the submitted job " + "graphs are not monitored (temporarily)."); case CONNECTION_LOST: LOG.warn("ZooKeeper connection LOST. Changes to the submitted job " + "graphs are not monitored (permanently)."); break; case CONNECTION_RECONNECTED: LOG.info("ZooKeeper connection RECONNECTED. Changes to the submitted job " + "graphs are monitored again."); case INITIALIZED: LOG.info("SubmittedJobGraphsPathCacheListener initialized"); break; {code} For CONNECTION_SUSPENDED and CONNECTION_RECONNECTED, the break statement is missing. This would result in unrelated event logging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-2963) Dependence on SerializationUtils#deserialize() should be avoided
Ted Yu created FLINK-2963: - Summary: Dependence on SerializationUtils#deserialize() should be avoided Key: FLINK-2963 URL: https://issues.apache.org/jira/browse/FLINK-2963 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor There is a problem with `SerializationUtils` from Apache Commons Lang. Here is an open issue where the class will throw a `ClassNotFoundException` even if the class is in the classpath in a multiple-classloader environment: https://issues.apache.org/jira/browse/LANG-1049 {code} state = (HashMap) SerializationUtils.deserialize(bais); ./flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/NonKeyedWindowOperator.java state = (HashMap) SerializationUtils.deserialize(bais); ./flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java return SerializationUtils.deserialize(message); ./flink-streaming-java/src/main/java/org/apache/flink/streaming/util/serialization/JavaDefaultStringSchema.java T copied = SerializationUtils.deserialize(SerializationUtils ./flink-streaming-java/src/test/java/org/apache/flink/streaming/util/MockOutput.java {code} We should move away from SerializationUtils.deserialize() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3005) Commons-collections object deserialization remote command execution vulnerability
Ted Yu created FLINK-3005: - Summary: Commons-collections object deserialization remote command execution vulnerability Key: FLINK-3005 URL: https://issues.apache.org/jira/browse/FLINK-3005 Project: Flink Issue Type: Bug Reporter: Ted Yu http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in-common-this-vulnerability/ TL;DR: If you have commons-collections on your classpath and accept and process Java object serialization data, then you may have an exploitable remote command execution vulnerability. Brief search in code base for ObjectInputStream reveals several places where the vulnerability exists. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3014) Return code from InputStream#read() should be checked in PythonStreamer#sendBroadCastVariables
Ted Yu created FLINK-3014: - Summary: Return code from InputStream#read() should be checked in PythonStreamer#sendBroadCastVariables Key: FLINK-3014 URL: https://issues.apache.org/jira/browse/FLINK-3014 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is related code (there're 3 occurrences in the method): {code} in.read(buffer, 0, 4); checkForError(); {code} java.io.InputStream.read(byte[], int, int) returns the number of bytes read. The return value should be checked because checkForError() does this: {code} if (getInt(buffer, 0) == -2) { try { //wait before terminating to ensure that the complete error message is printed {code} Incorrect result may be returned if fewer than 4 bytes of data are read. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3103) Accessing closed in FsStateBackend#FsCheckpointStateOutputStream#flush() should be synchronized
Ted Yu created FLINK-3103: - Summary: Accessing closed in FsStateBackend#FsCheckpointStateOutputStream#flush() should be synchronized Key: FLINK-3103 URL: https://issues.apache.org/jira/browse/FLINK-3103 Project: Flink Issue Type: Bug Reporter: Ted Yu Here is related code: {code} public void flush() throws IOException { if (!closed) { {code} Lock "FsCheckpointStateOutputStream.this" should be held. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3104) Potential null dereference of outStream in FsStateBackend#CheckpointStateOutputStream#closeAndGetHandle()
Ted Yu created FLINK-3104: - Summary: Potential null dereference of outStream in FsStateBackend#CheckpointStateOutputStream#closeAndGetHandle() Key: FLINK-3104 URL: https://issues.apache.org/jira/browse/FLINK-3104 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is related code: {code} if (outStream == null && pos <= localStateThreshold) { closed = true; byte[] bytes = Arrays.copyOf(writeBuffer, pos); return new ByteStreamStateHandle(bytes); } else { flush(); outStream.close(); {code} The if condition checks both outStream and pos. In the else block, outStream may be null, leading to NPE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3137) Missing break in OutputEmitter ctor
Ted Yu created FLINK-3137: - Summary: Missing break in OutputEmitter ctor Key: FLINK-3137 URL: https://issues.apache.org/jira/browse/FLINK-3137 Project: Flink Issue Type: Bug Reporter: Ted Yu Here is related code: {code} switch (strategy) { case PARTITION_CUSTOM: extractedKeys = new Object[1]; case FORWARD: case PARTITION_HASH: {code} Looks like a break is missing following assignment to extractedKeys -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3196) InputStream should be closed in EnvironmentInformation#getRevisionInformation()
Ted Yu created FLINK-3196: - Summary: InputStream should be closed in EnvironmentInformation#getRevisionInformation() Key: FLINK-3196 URL: https://issues.apache.org/jira/browse/FLINK-3196 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is related code: {code} InputStream propFile = EnvironmentInformation.class.getClassLoader().getResourceAsStream(".version.properties"); if (propFile != null) { Properties properties = new Properties(); properties.load(propFile); {code} propFile should be closed upon leaving the method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3197) InputStream not closed in BinaryInputFormat#createStatistics
Ted Yu created FLINK-3197: - Summary: InputStream not closed in BinaryInputFormat#createStatistics Key: FLINK-3197 URL: https://issues.apache.org/jira/browse/FLINK-3197 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is related code: {code} FSDataInputStream fdis = file.getPath().getFileSystem().open(file.getPath(), blockInfo.getInfoSize()); fdis.seek(file.getLen() - blockInfo.getInfoSize()); blockInfo.read(new DataInputViewStreamWrapper(fdis)); totalCount += blockInfo.getAccumulatedRecordCount(); {code} fdis / wrapper should be closed upon leaving the method -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3210) Unnecessary call to deserializer#deserialize() in LegacyFetcher#SimpleConsumerThread#run()
Ted Yu created FLINK-3210: - Summary: Unnecessary call to deserializer#deserialize() in LegacyFetcher#SimpleConsumerThread#run() Key: FLINK-3210 URL: https://issues.apache.org/jira/browse/FLINK-3210 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is related code: {code} byte[] valueBytes; if (payload == null) { deletedMessages++; valueBytes = null; } else { ... final T value = deserializer.deserialize(keyBytes, valueBytes, fp.topic, offset); {code} When valueBytes is null, there is no need to call deserializer#deserialize() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3222) Incorrect shift amount in OperatorCheckpointStats#hashCode()
Ted Yu created FLINK-3222: - Summary: Incorrect shift amount in OperatorCheckpointStats#hashCode() Key: FLINK-3222 URL: https://issues.apache.org/jira/browse/FLINK-3222 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is related code: {code} result = 31 * result + (int) (subTaskStats.length ^ (subTaskStats.length >>> 32)); {code} subTaskStats.length is an int. The shift amount is greater than 31 bits. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3269) Return value from DataInputStream#read() should be checked in PythonPlanReceiver
Ted Yu created FLINK-3269: - Summary: Return value from DataInputStream#read() should be checked in PythonPlanReceiver Key: FLINK-3269 URL: https://issues.apache.org/jira/browse/FLINK-3269 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor There're several places where the return value from DataInputStream#read() isn't checked. Here is an example from BytesDeserializer : {code} byte[] buffer = new byte[size]; input.read(buffer); return buffer; {code} Buffer of size bytes is allocated. However, the read operation may not fill size bytes. The return value should be checked. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3280) Wrong usage of Boolean.getBoolean()
Ted Yu created FLINK-3280: - Summary: Wrong usage of Boolean.getBoolean() Key: FLINK-3280 URL: https://issues.apache.org/jira/browse/FLINK-3280 Project: Flink Issue Type: Bug Reporter: Ted Yu I observed the problem in both FlinkKafkaProducerBase.java and FlinkKafkaConsumer09.java >From FlinkKafkaProducerBase.java : {code} if(!Boolean.getBoolean(producerConfig.getProperty(KEY_DISABLE_METRICS, "false"))) { {code} Boolean.getBoolean() would call System.getProperty(). See http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/lang/Boolean.java#238 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3301) Ineffective synchronization in MessageAcknowledgingSourceBase#restoreState
Ted Yu created FLINK-3301: - Summary: Ineffective synchronization in MessageAcknowledgingSourceBase#restoreState Key: FLINK-3301 URL: https://issues.apache.org/jira/browse/FLINK-3301 Project: Flink Issue Type: Bug Reporter: Ted Yu Here is related code: {code} public void restoreState(SerializedCheckpointData[] state) throws Exception { synchronized (pendingCheckpoints) { pendingCheckpoints = SerializedCheckpointData.toDeque(state, idSerializer); {code} When lock is held on current pendingCheckpoints field, new reference is assigned to the same field - making the synchronization ineffective. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3325) Unclosed InputStream in CEPPatternOperator#restoreState()
Ted Yu created FLINK-3325: - Summary: Unclosed InputStream in CEPPatternOperator#restoreState() Key: FLINK-3325 URL: https://issues.apache.org/jira/browse/FLINK-3325 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is related code: {code} final InputStream is = stream.getState(getUserCodeClassloader()); final ObjectInputStream ois = new ObjectInputStream(is); final DataInputViewStreamWrapper div = new DataInputViewStreamWrapper(is); nfa = (NFA)ois.readObject(); {code} Neither is nor ois is closed upon return from the method -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3329) Unclosed BackupEngine in AbstractRocksDBState#snapshot()
Ted Yu created FLINK-3329: - Summary: Unclosed BackupEngine in AbstractRocksDBState#snapshot() Key: FLINK-3329 URL: https://issues.apache.org/jira/browse/FLINK-3329 Project: Flink Issue Type: Bug Reporter: Ted Yu Here is related code: {code} BackupEngine backupEngine = BackupEngine.open(Env.getDefault(), new BackupableDBOptions(localBackupPath.getAbsolutePath())); backupEngine.createNewBackup(db); {code} BackupEngine implements AutoCloseable. backupEngine should be closed upon return from the method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3362) Potential overflow in NettyBufferPool#getNumberOfAllocatedBytes
Ted Yu created FLINK-3362: - Summary: Potential overflow in NettyBufferPool#getNumberOfAllocatedBytes Key: FLINK-3362 URL: https://issues.apache.org/jira/browse/FLINK-3362 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is related code: long allocatedBytes = numChunks * chunkSize; The evaluation of numChunks * chunkSize is done using 32-bit arithmetic. This may result in overflow since the expected type of expression is long. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3392) Unprotected access to elements in ClosableBlockingQueue#size()
Ted Yu created FLINK-3392: - Summary: Unprotected access to elements in ClosableBlockingQueue#size() Key: FLINK-3392 URL: https://issues.apache.org/jira/browse/FLINK-3392 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is related code: {code} public int size() { return elements.size(); } {code} Access to elements should be protected by lock.lock() / lock.unlock() pair. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3433) Return value from ObjectInputStream#read() should be checked in AvroOutputFormat#readObject
Ted Yu created FLINK-3433: - Summary: Return value from ObjectInputStream#read() should be checked in AvroOutputFormat#readObject Key: FLINK-3433 URL: https://issues.apache.org/jira/browse/FLINK-3433 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is related code: {code} if(length != 0) { byte[] json = new byte[length]; in.read(json); Schema schema = new Schema.Parser().parse(new String(json)); {code} The return value from in.read() should be verified against length. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3434) Return value from flinkYarnClient#deploy() should be checked against null
Ted Yu created FLINK-3434: - Summary: Return value from flinkYarnClient#deploy() should be checked against null Key: FLINK-3434 URL: https://issues.apache.org/jira/browse/FLINK-3434 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor In CliFrontend#getClient() : {code} yarnCluster = flinkYarnClient.deploy(); yarnCluster.connectToCluster(); {code} Looking at FlinkYarnClientBase#deployInternal(), null return is possible: {code} try { org.apache.flink.core.fs.FileSystem.setDefaultScheme(flinkConfiguration); } catch (IOException e) { LOG.error("Error while setting the default " + "filesystem scheme from configuration.", e); return null; } {code} We should check the return from deploy() before calling connectToCluster(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3525) Missing call to super#close() in TimestampsAndPeriodicWatermarksOperator#close()
Ted Yu created FLINK-3525: - Summary: Missing call to super#close() in TimestampsAndPeriodicWatermarksOperator#close() Key: FLINK-3525 URL: https://issues.apache.org/jira/browse/FLINK-3525 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is related code: {code} public void close() throws Exception { // emit a final watermark {code} AbstractUdfStreamOperator#close() should be called. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3576) Upgrade Snappy Java to 1.1.2.1
Ted Yu created FLINK-3576: - Summary: Upgrade Snappy Java to 1.1.2.1 Key: FLINK-3576 URL: https://issues.apache.org/jira/browse/FLINK-3576 Project: Flink Issue Type: Improvement Reporter: Ted Yu Priority: Minor There was a JVM memory leaky problem reported in https://github.com/xerial/snappy-java/issues/131 The above issue has been resolved. 1.1.2.1 was released on Jan 22nd. We should upgrade to this release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3635) Potential null deference in TwitterExample#SelectEnglishAndTokenizeFlatMap#flatMap
Ted Yu created FLINK-3635: - Summary: Potential null deference in TwitterExample#SelectEnglishAndTokenizeFlatMap#flatMap Key: FLINK-3635 URL: https://issues.apache.org/jira/browse/FLINK-3635 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} JsonNode jsonNode = jsonParser.readValue(value, JsonNode.class); if (jsonNode.has("user") && jsonNode.get("user").get("lang").asText().equals("en")) { {code} jsonNode.get("user").get("lang") may return null, leading to NullPointerException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3668) Potential null deference in HadoopInputFormatBase#createInputSplits()
Ted Yu created FLINK-3668: - Summary: Potential null deference in HadoopInputFormatBase#createInputSplits() Key: FLINK-3668 URL: https://issues.apache.org/jira/browse/FLINK-3668 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is the code: {code} List splits; try { splits = this.mapreduceInputFormat.getSplits(jobContext); } catch (InterruptedException e) { throw new IOException("Could not get Splits.", e); } HadoopInputSplit[] hadoopInputSplits = new HadoopInputSplit[splits.size()]; {code} If InterruptedException is caught, splits would be null. Yet, the next line accesses splits.size() without null check. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3732) Potential null deference in ExecutionConfig#equals()
Ted Yu created FLINK-3732: - Summary: Potential null deference in ExecutionConfig#equals() Key: FLINK-3732 URL: https://issues.apache.org/jira/browse/FLINK-3732 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} ((restartStrategyConfiguration == null && other.restartStrategyConfiguration == null) || restartStrategyConfiguration.equals(other.restartStrategyConfiguration)) && {code} If restartStrategyConfiguration is null but other.restartStrategyConfiguration is not null, the above would result in NPE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3733) registeredTypesWithKryoSerializers is not assigned in ExecutionConfig#deserializeUserCode()
Ted Yu created FLINK-3733: - Summary: registeredTypesWithKryoSerializers is not assigned in ExecutionConfig#deserializeUserCode() Key: FLINK-3733 URL: https://issues.apache.org/jira/browse/FLINK-3733 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} if (serializedRegisteredTypesWithKryoSerializers != null) { registeredTypesWithKryoSerializers = serializedRegisteredTypesWithKryoSerializers.deserializeValue(userCodeClassLoader); } else { registeredTypesWithKryoSerializerClasses = new LinkedHashMap<>(); } {code} When serializedRegisteredTypesWithKryoSerializers is null, registeredTypesWithKryoSerializers is not assigned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3734) Unclosed DataInputView in AbstractAlignedProcessingTimeWindowOperator#restoreState()
Ted Yu created FLINK-3734: - Summary: Unclosed DataInputView in AbstractAlignedProcessingTimeWindowOperator#restoreState() Key: FLINK-3734 URL: https://issues.apache.org/jira/browse/FLINK-3734 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} DataInputView in = inputState.getState(getUserCodeClassloader()); final long nextEvaluationTime = in.readLong(); final long nextSlideTime = in.readLong(); AbstractKeyedTimePanes panes = createPanes(keySelector, function); panes.readFromInput(in, keySerializer, stateTypeSerializer); restoredState = new RestoredState<>(panes, nextEvaluationTime, nextSlideTime); } {code} DataInputView in is not closed upon return. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3753) KillerWatchDog should not use kill on toKill thread
Ted Yu created FLINK-3753: - Summary: KillerWatchDog should not use kill on toKill thread Key: FLINK-3753 URL: https://issues.apache.org/jira/browse/FLINK-3753 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} // this is harsh, but this watchdog is a last resort if (toKill.isAlive()) { toKill.stop(); } {code} stop() is deprecated. See: https://www.securecoding.cert.org/confluence/display/java/THI05-J.+Do+not+use+Thread.stop()+to+terminate+threads -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3773) Scanners are left unclosed in SqlExplainTest
Ted Yu created FLINK-3773: - Summary: Scanners are left unclosed in SqlExplainTest Key: FLINK-3773 URL: https://issues.apache.org/jira/browse/FLINK-3773 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor e.g. {code} String source = new Scanner(new File(testFilePath + "../../src/test/scala/resources/testFilter0.out")) {code} Scanner implements AutoCloseable. Using try-with-resources would be a good pattern for closing the Scanners. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3781) BlobClient may be left unclosed in BlobCache#deleteGlobal()
Ted Yu created FLINK-3781: - Summary: BlobClient may be left unclosed in BlobCache#deleteGlobal() Key: FLINK-3781 URL: https://issues.apache.org/jira/browse/FLINK-3781 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} public void deleteGlobal(BlobKey key) throws IOException { delete(key); BlobClient bc = createClient(); bc.delete(key); bc.close(); {code} If delete() throws IOException, BlobClient would be left inclosed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3801) Upgrade Joda-Time library to 2.9.3
Ted Yu created FLINK-3801: - Summary: Upgrade Joda-Time library to 2.9.3 Key: FLINK-3801 URL: https://issues.apache.org/jira/browse/FLINK-3801 Project: Flink Issue Type: Improvement Reporter: Ted Yu Priority: Minor Currently yoda-time 2.5 is used which was very old. We should upgrade to 2.9.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3809) Missing break in ZooKeeperLeaderRetrievalService#handleStateChange()
Ted Yu created FLINK-3809: - Summary: Missing break in ZooKeeperLeaderRetrievalService#handleStateChange() Key: FLINK-3809 URL: https://issues.apache.org/jira/browse/FLINK-3809 Project: Flink Issue Type: Bug Reporter: Ted Yu {code} protected void handleStateChange(ConnectionState newState) { switch (newState) { case CONNECTED: LOG.debug("Connected to ZooKeeper quorum. Leader retrieval can start."); case SUSPENDED: LOG.warn("Connection to ZooKeeper suspended. Can no longer retrieve the leader from " + "ZooKeeper."); case RECONNECTED: LOG.info("Connection to ZooKeeper was reconnected. Leader retrieval can be restarted."); case LOST: LOG.warn("Connection to ZooKeeper lost. Can no longer retrieve the leader from " + "ZooKeeper."); } {code} Except for LOST state, the other states would lead to multiple logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3810) Missing break in ZooKeeperLeaderElectionService#handleStateChange()
Ted Yu created FLINK-3810: - Summary: Missing break in ZooKeeperLeaderElectionService#handleStateChange() Key: FLINK-3810 URL: https://issues.apache.org/jira/browse/FLINK-3810 Project: Flink Issue Type: Bug Reporter: Ted Yu {code} protected void handleStateChange(ConnectionState newState) { switch (newState) { case CONNECTED: LOG.debug("Connected to ZooKeeper quorum. Leader election can start."); case SUSPENDED: LOG.warn("Connection to ZooKeeper suspended. The contender " + leaderContender.getAddress() + "no longer participates in the leader election."); case RECONNECTED: LOG.info("Connection to ZooKeeper was reconnected. Leader election can be restarted."); case LOST: // Maybe we have to throw an exception here to terminate the JobManager LOG.warn("Connection to ZooKeeper lost. The contender " + leaderContender.getAddress() + "no longer participates in the leader election."); {code} Any of the first 3 states would result in multiple log output. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3928) Potential overflow due to 32-bit int arithmetic
Ted Yu created FLINK-3928: - Summary: Potential overflow due to 32-bit int arithmetic Key: FLINK-3928 URL: https://issues.apache.org/jira/browse/FLINK-3928 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor The following pattern occurs in both LocalClusteringCoefficient.java and TriangleListing.java : {code} int scale = parameters.getInt("scale", DEFAULT_SCALE); ... long vertexCount = 1 << scale; {code} "1 << scale" may overflow with type "int" due to the use of 32-bit arithmetic -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3958) Access to MetricRegistry doesn't have proper synchronization in some classes
Ted Yu created FLINK-3958: - Summary: Access to MetricRegistry doesn't have proper synchronization in some classes Key: FLINK-3958 URL: https://issues.apache.org/jira/browse/FLINK-3958 Project: Flink Issue Type: Bug Reporter: Ted Yu In GraphiteReporter#getReporter(): {code} com.codahale.metrics.graphite.GraphiteReporter.Builder builder = com.codahale.metrics.graphite.GraphiteReporter.forRegistry(registry); {code} Access to registry should be protected by lock on ScheduledDropwizardReporter.this Similar issue exists in GangliaReporter#getReporter() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3998) Access to gauges and counters in StatsDReporter#report() is not properly synchronized
Ted Yu created FLINK-3998: - Summary: Access to gauges and counters in StatsDReporter#report() is not properly synchronized Key: FLINK-3998 URL: https://issues.apache.org/jira/browse/FLINK-3998 Project: Flink Issue Type: Bug Reporter: Ted Yu {code} for (Map.Entry, String> entry : gauges.entrySet()) { reportGauge(entry.getValue(), entry.getKey()); } for (Map.Entry entry : counters.entrySet()) { reportCounter(entry.getValue(), entry.getKey()); {code} Access to gauges and counters should be protected by lock on AbstractReporter.this -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4053) Return value from Connection should be checked against null
Ted Yu created FLINK-4053: - Summary: Return value from Connection should be checked against null Key: FLINK-4053 URL: https://issues.apache.org/jira/browse/FLINK-4053 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor In RMQSource.java and RMQSink.java, there is code in the following pattern: {code} connection = factory.newConnection(); channel = connection.createChannel(); {code} According to https://www.rabbitmq.com/releases/rabbitmq-java-client/current-javadoc/com/rabbitmq/client/Connection.html#createChannel() : {code} Returns: a new channel descriptor, or null if none is available {code} The return value should be checked against null. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4076) BoltWrapper#dispose() should call AbstractStreamOperator#dispose()
Ted Yu created FLINK-4076: - Summary: BoltWrapper#dispose() should call AbstractStreamOperator#dispose() Key: FLINK-4076 URL: https://issues.apache.org/jira/browse/FLINK-4076 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} @Override public void dispose() { this.bolt.cleanup(); } {code} AbstractStreamOperator#dispose() should be called first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4089) Ineffective null check in YarnClusterClient#getApplicationStatus()
Ted Yu created FLINK-4089: - Summary: Ineffective null check in YarnClusterClient#getApplicationStatus() Key: FLINK-4089 URL: https://issues.apache.org/jira/browse/FLINK-4089 Project: Flink Issue Type: Bug Reporter: Ted Yu Here is related code: {code} if(pollingRunner == null) { LOG.warn("YarnClusterClient.getApplicationStatus() has been called on an uninitialized cluster." + "The system might be in an erroneous state"); } ApplicationReport lastReport = pollingRunner.getLastReport(); {code} If pollingRunner is null, NPE would result from the getLastReport() call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4090) Close of OutputStream should be in finally clause in FlinkYarnSessionCli#writeYarnProperties()
Ted Yu created FLINK-4090: - Summary: Close of OutputStream should be in finally clause in FlinkYarnSessionCli#writeYarnProperties() Key: FLINK-4090 URL: https://issues.apache.org/jira/browse/FLINK-4090 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} try { OutputStream out = new FileOutputStream(propertiesFile); properties.store(out, "Generated YARN properties file"); out.close(); } catch (IOException e) { {code} The close of out should be in finally cluase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4096) Nested FileOutputStream is not closed in JarFileCreator
Ted Yu created FLINK-4096: - Summary: Nested FileOutputStream is not closed in JarFileCreator Key: FLINK-4096 URL: https://issues.apache.org/jira/browse/FLINK-4096 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} final JarOutputStream jos = new JarOutputStream(new FileOutputStream(this.outputFile), new Manifest()); {code} When jos is closed, the FileOutputStream is not automatically closed. FileOutputStream should be closed explicitly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4258) Potential null pointer dereference in SavepointCoordinator#onFullyAcknowledgedCheckpoint
Ted Yu created FLINK-4258: - Summary: Potential null pointer dereference in SavepointCoordinator#onFullyAcknowledgedCheckpoint Key: FLINK-4258 URL: https://issues.apache.org/jira/browse/FLINK-4258 Project: Flink Issue Type: Bug Reporter: Ted Yu {code} if (promise == null) { LOG.info("Pending savepoint with ID " + checkpoint.getCheckpointID() + " has been " + "removed before receiving acknowledgment."); } // Sanity check if (promise.isCompleted()) { throw new IllegalStateException("Savepoint promise completed"); {code} Looks like a return statement is missing in the first if block above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4259) Unclosed FSDataOutputStream in FileCache#copy()
Ted Yu created FLINK-4259: - Summary: Unclosed FSDataOutputStream in FileCache#copy() Key: FLINK-4259 URL: https://issues.apache.org/jira/browse/FLINK-4259 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} try { FSDataOutputStream lfsOutput = tFS.create(targetPath, false); FSDataInputStream fsInput = sFS.open(sourcePath); IOUtils.copyBytes(fsInput, lfsOutput); {code} The FSDataOutputStream lfsOutput should be closed upon exit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4309) Potential null pointer dereference in DelegatingConfiguration#keySet()
Ted Yu created FLINK-4309: - Summary: Potential null pointer dereference in DelegatingConfiguration#keySet() Key: FLINK-4309 URL: https://issues.apache.org/jira/browse/FLINK-4309 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} final int prefixLen = this.prefix == null ? 0 : this.prefix.length(); for (String key : this.backingConfig.keySet()) { if (key.startsWith(this.prefix)) { {code} If this.prefix == null, we would get NPE in startsWith(): {code} public boolean startsWith(String prefix, int toffset) { char ta[] = value; int to = toffset; char pa[] = prefix.value; {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4436) Unclosed DataOutputBuffer in Utils#setTokensFor()
Ted Yu created FLINK-4436: - Summary: Unclosed DataOutputBuffer in Utils#setTokensFor() Key: FLINK-4436 URL: https://issues.apache.org/jira/browse/FLINK-4436 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} DataOutputBuffer dob = new DataOutputBuffer(); credentials.writeTokenStorageToStream(dob); {code} dob should be closed upon returning from the method. YarnApplicationMasterRunner#createTaskManagerContext() has similar issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4437) Lock evasion around lastTriggeredCheckpoint may lead to lost updates to related fields
Ted Yu created FLINK-4437: - Summary: Lock evasion around lastTriggeredCheckpoint may lead to lost updates to related fields Key: FLINK-4437 URL: https://issues.apache.org/jira/browse/FLINK-4437 Project: Flink Issue Type: Bug Reporter: Ted Yu In CheckpointCoordinator#triggerCheckpoint(): {code} // make sure the minimum interval between checkpoints has passed if (lastTriggeredCheckpoint + minPauseBetweenCheckpoints > timestamp) { {code} If two threads evaluate 'lastTriggeredCheckpoint + minPauseBetweenCheckpoints > timestamp' in close proximity before lastTriggeredCheckpoint is updated, the two threads may have an inconsistent view of "lastTriggeredCheckpoint" and updates to fields correlated with "lastTriggeredCheckpoint" may be lost. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4482) numUnsuccessfulCheckpointsTriggers is accessed without holding triggerLock
Ted Yu created FLINK-4482: - Summary: numUnsuccessfulCheckpointsTriggers is accessed without holding triggerLock Key: FLINK-4482 URL: https://issues.apache.org/jira/browse/FLINK-4482 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor In CheckpointCoordinator#stopCheckpointScheduler() : {code} synchronized (lock) { ... numUnsuccessfulCheckpointsTriggers = 0; {code} triggerLock is not held in the above operation. See comment for triggerLock earlier in triggerCheckpoint(): {code} // we lock with a special lock to make sure that trigger requests do not overtake each other. // this is not done with the coordinator-wide lock, because the 'checkpointIdCounter' // may issue blocking operations. Using a different lock than teh coordinator-wide lock, // we avoid blocking the processing of 'acknowledge/decline' messages during that time. synchronized (triggerLock) { {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4499) Introduce findbugs maven plugin
Ted Yu created FLINK-4499: - Summary: Introduce findbugs maven plugin Key: FLINK-4499 URL: https://issues.apache.org/jira/browse/FLINK-4499 Project: Flink Issue Type: Improvement Reporter: Ted Yu As suggested by Stephan in FLINK-4482, this issue is to add findbugs-maven-plugin into the build process so that we can detect lack of proper locking and other defects automatically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4531) Thread.stop() is deprecated
Ted Yu created FLINK-4531: - Summary: Thread.stop() is deprecated Key: FLINK-4531 URL: https://issues.apache.org/jira/browse/FLINK-4531 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor In KillerWatchDog#run() : {code} if (toKill.isAlive()) { toKill.stop(); } {code} Thread.stop() is deprecated. https://www.securecoding.cert.org/confluence/display/java/THI05-J.+Do+not+use+Thread.stop()+to+terminate+threads -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4533) Unprotected access to meters in StatsDReporter#report()
Ted Yu created FLINK-4533: - Summary: Unprotected access to meters in StatsDReporter#report() Key: FLINK-4533 URL: https://issues.apache.org/jira/browse/FLINK-4533 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Access to meters in AbstractReporter is protected by AbstractReporter.this. Access to meters in StatsDReporter#report() should do the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4534) Lack of synchronization in BucketingSink#restoreState() w.r.t. state.bucketStates
Ted Yu created FLINK-4534: - Summary: Lack of synchronization in BucketingSink#restoreState() w.r.t. state.bucketStates Key: FLINK-4534 URL: https://issues.apache.org/jira/browse/FLINK-4534 Project: Flink Issue Type: Bug Reporter: Ted Yu Iteration over state.bucketStates is protected by synchronization in other methods, except for the following in restoreState(): {code} for (BucketState bucketState : state.bucketStates.values()) { {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-6997) SavepointITCase fails in master branch
Ted Yu created FLINK-6997: - Summary: SavepointITCase fails in master branch Key: FLINK-6997 URL: https://issues.apache.org/jira/browse/FLINK-6997 Project: Flink Issue Type: Bug Components: Tests Reporter: Ted Yu I got the following test failure (with commit a0b781461bcf8c2f1d00b93464995f03eda589f1) {code} testSavepointForJobWithIteration(org.apache.flink.test.checkpointing.SavepointITCase) Time elapsed: 8.129 sec <<< ERROR! java.io.IOException: java.lang.Exception: Failed to complete savepoint at org.apache.flink.runtime.testingUtils.TestingCluster.triggerSavepoint(TestingCluster.scala:342) at org.apache.flink.runtime.testingUtils.TestingCluster.triggerSavepoint(TestingCluster.scala:316) at org.apache.flink.test.checkpointing.SavepointITCase.testSavepointForJobWithIteration(SavepointITCase.java:827) Caused by: java.lang.Exception: Failed to complete savepoint at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anon$7.apply(JobManager.scala:821) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anon$7.apply(JobManager.scala:805) at org.apache.flink.runtime.concurrent.impl.FlinkFuture$5.onComplete(FlinkFuture.java:272) at akka.dispatch.OnComplete.internal(Future.scala:247) at akka.dispatch.OnComplete.internal(Future.scala:245) at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175) at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91) at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91) at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.lang.Exception: Failed to trigger savepoint: Not all required tasks are currently running. at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.triggerSavepoint(CheckpointCoordinator.java:382) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:800) at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) at org.apache.flink.runtime.testingUtils.TestingJobManagerLike$$anonfun$handleTestingMessage$1.applyOrElse(TestingJobManagerLike.scala:95) at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162) at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:38) at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33) at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28) at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7049) TestingApplicationMaster keeps running after integration tests finish
Ted Yu created FLINK-7049: - Summary: TestingApplicationMaster keeps running after integration tests finish Key: FLINK-7049 URL: https://issues.apache.org/jira/browse/FLINK-7049 Project: Flink Issue Type: Test Reporter: Ted Yu Priority: Minor After integration tests finish, TestingApplicationMaster is still running. Toward the end of flink-yarn-tests/target/flink-yarn-tests-ha/flink-yarn-tests-ha-logDir-nm-1_0/application_1498768839874_0001/container_1498768839874_0001_03_01/jobmanager.log : {code} 2017-06-29 22:09:49,681 INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:46165 2017-06-29 22:09:49,681 ERROR org.apache.flink.shaded.org.apache.curator.ConnectionState- Authentication failed 2017-06-29 22:09:49,682 WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 2017-06-29 22:09:50,782 WARN org.apache.zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-3597644653611245612.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. 2017-06-29 22:09:50,782 INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:46165 2017-06-29 22:09:50,782 ERROR org.apache.flink.shaded.org.apache.curator.ConnectionState- Authentication failed 2017-06-29 22:09:50,783 WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7197) Missing call to GraphAlgorithmWrappingBase#canMergeConfigurationWith()
Ted Yu created FLINK-7197: - Summary: Missing call to GraphAlgorithmWrappingBase#canMergeConfigurationWith() Key: FLINK-7197 URL: https://issues.apache.org/jira/browse/FLINK-7197 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor In flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/asm/translate/TranslateVertexValues.java : {code} protected boolean canMergeConfigurationWith(GraphAlgorithmWrappingBase other) { super.mergeConfiguration(other); {code} GraphAlgorithmWrappingBase#canMergeConfigurationWith() should be called. There is similar issue in TranslateGraphIds.java -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7260) Match not exhaustive in WindowJoinUtil.scala
Ted Yu created FLINK-7260: - Summary: Match not exhaustive in WindowJoinUtil.scala Key: FLINK-7260 URL: https://issues.apache.org/jira/browse/FLINK-7260 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is related code: {code} [WARNING] /mnt/disk2/a/flink/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/join/WindowJoinUtil.scala:296: warning: match may not be exhaustive. [WARNING] It would fail on the following inputs: BETWEEN, CREATE_VIEW, DEFAULT, DESCRIBE_SCHEMA, DOT, EXCEPT, EXTRACT, GREATEST, LAST_VALUE, MAP_QUERY_CONSTRUCTOR, NOT, NOT_EQUALS, NULLS_LAST, PATTERN_ALTER, PREV, SIMILAR, SUM0, TIMESTAMP_DIFF, UNION, VAR_SAMP [WARNING] timePred.pred.getKind match { {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7402) Ineffective null check in NettyMessage#write()
Ted Yu created FLINK-7402: - Summary: Ineffective null check in NettyMessage#write() Key: FLINK-7402 URL: https://issues.apache.org/jira/browse/FLINK-7402 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is the null check in finally block: {code} finally { if (buffer != null) { buffer.recycle(); } {code} But buffer has been dereferenced in the try block without guard. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7432) Unclosed HighAvailabilityServices instance in QueryableStateClient
Ted Yu created FLINK-7432: - Summary: Unclosed HighAvailabilityServices instance in QueryableStateClient Key: FLINK-7432 URL: https://issues.apache.org/jira/browse/FLINK-7432 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} public QueryableStateClient(Configuration config) throws Exception { this(config, HighAvailabilityServicesUtils.createHighAvailabilityServices( config, Executors.directExecutor(), HighAvailabilityServicesUtils.AddressResolution.TRY_ADDRESS_RESOLUTION)); {code} The HighAvailabilityServices instance is only used for calling getJobManagerLeaderRetriever(). The instance should be closed upon leaving QueryableStateClient ctor. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7433) Lack of synchronization accessing inProgress in JobCancellationWithSavepointHandlers
Ted Yu created FLINK-7433: - Summary: Lack of synchronization accessing inProgress in JobCancellationWithSavepointHandlers Key: FLINK-7433 URL: https://issues.apache.org/jira/browse/FLINK-7433 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} try { if (throwable != null) { completed.put(requestId, throwable); } else { completed.put(requestId, path); } } finally { inProgress.remove(jobId); } {code} The call to inProgress.remove(jobId) should be protected by lock object. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7437) Upgrade calcite to calcite 1.14
Ted Yu created FLINK-7437: - Summary: Upgrade calcite to calcite 1.14 Key: FLINK-7437 URL: https://issues.apache.org/jira/browse/FLINK-7437 Project: Flink Issue Type: Task Reporter: Ted Yu The upcoming calcite 1.14 contains important fixes. e.g. CALCITE-1859 would allow HBaseConnectorITCase to pass. This issue is to upgrade Calcite to 1.14 release -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7438) Some classes are eclipsed by classes in package scala
Ted Yu created FLINK-7438: - Summary: Some classes are eclipsed by classes in package scala Key: FLINK-7438 URL: https://issues.apache.org/jira/browse/FLINK-7438 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Noticed the following during compilation: {code} [WARNING] /flink/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/WindowedStream.scala:33: warning: imported `OutputTag' is permanently hidden by definition of object OutputTag in package scala [WARNING] import org.apache.flink.util.{Collector, OutputTag} [WARNING] ^ [WARNING] /flink/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/WindowedStream.scala:33: warning: imported `OutputTag' is permanently hidden by definition of class OutputTag in package scala [WARNING] import org.apache.flink.util.{Collector, OutputTag} {code} We should avoid the warning e.r.t. OutputTag. There may be other occurrences of similar warning. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7488) TaskManagerHeapSizeCalculationJavaBashTest sometimes fails
Ted Yu created FLINK-7488: - Summary: TaskManagerHeapSizeCalculationJavaBashTest sometimes fails Key: FLINK-7488 URL: https://issues.apache.org/jira/browse/FLINK-7488 Project: Flink Issue Type: Test Reporter: Ted Yu Priority: Minor {code} TaskManagerHeapSizeCalculationJavaBashTest.compareHeapSizeShellScriptWithJava:110->compareHeapSizeJavaVsScript:275 Different heap sizes with configuration: {taskmanager.network.memory.fraction=0.1, taskmanager.memory.off-heap=false, taskmanager.memory.fraction=0.7, taskmanager.memory.size=-1, taskmanager.network.memory.max=1073741824, taskmanager.heap.mb=1000, taskmanager.network.memory.min=67108864} expected:<[]1000> but was:<[Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set.Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/hdp/2.5.0.0-1245/hadoop/conf:/usr/hdp/2.5.0.0-1245/hadoop/lib/*:/usr/hdp/2.5.0.0-1245/hadoop/.//*:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/./:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/.//*:/usr/hdp/2.5.0.0-1245/hadoop-yarn/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-yarn/.//*:/usr/hdp/2.5.0.0-1245/hadoop-mapreduce/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-mapreduce/.//*:/usr/hdp/2.5.0.0-1245/tez/*:/usr/hdp/2.5.0.0-1245/tez/lib/*:/usr/hdp/2.5.0.0-1245/tez/conf]1000> TaskManagerHeapSizeCalculationJavaBashTest.compareNetworkBufShellScriptWithJava:81->compareNetworkBufJavaVsScript:235 Different network buffer memory sizes with configuration: {taskmanager.network.memory.fraction=0.1, taskmanager.memory.off-heap=false, taskmanager.memory.fraction=0.7, taskmanager.memory.size=-1, taskmanager.network.memory.max=1073741824, taskmanager.heap.mb=1000, taskmanager.network.memory.min=67108864} expected:<[]104857600> but was:<[Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set.Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/hdp/2.5.0.0-1245/hadoop/conf:/usr/hdp/2.5.0.0-1245/hadoop/lib/*:/usr/hdp/2.5.0.0-1245/hadoop/.//*:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/./:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-hdfs/.//*:/usr/hdp/2.5.0.0-1245/hadoop-yarn/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-yarn/.//*:/usr/hdp/2.5.0.0-1245/hadoop-mapreduce/lib/*:/usr/hdp/2.5.0.0-1245/hadoop-mapreduce/.//*:/usr/hdp/2.5.0.0-1245/tez/*:/usr/hdp/2.5.0.0-1245/tez/lib/*:/usr/hdp/2.5.0.0-1245/tez/conf]104857600> {code} $HADOOP_CONF_DIR was not set prior to running the test. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7495) AbstractUdfStreamOperator#initializeState() should be called in AsyncWaitOperator#initializeState()
Ted Yu created FLINK-7495: - Summary: AbstractUdfStreamOperator#initializeState() should be called in AsyncWaitOperator#initializeState() Key: FLINK-7495 URL: https://issues.apache.org/jira/browse/FLINK-7495 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} recoveredStreamElements = context .getOperatorStateStore() .getListState(new ListStateDescriptor<>(STATE_NAME, inStreamElementSerializer)); {code} Call to AbstractUdfStreamOperator#initializeState() should be added in the beginning -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7525) Add config option to disable Cancel functionality on UI
Ted Yu created FLINK-7525: - Summary: Add config option to disable Cancel functionality on UI Key: FLINK-7525 URL: https://issues.apache.org/jira/browse/FLINK-7525 Project: Flink Issue Type: Improvement Reporter: Ted Yu In this email thread http://search-hadoop.com/m/Flink/VkLeQlf0QOnc7YA?subj=Security+Control+of+running+Flink+Jobs+on+Flink+UI , Raja was asking for a way to control the way users cancel Job(s). Robert proposed adding a config option which disables the Cancel functionality. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7588) Document RocksDB tuning for spinning disks
Ted Yu created FLINK-7588: - Summary: Document RocksDB tuning for spinning disks Key: FLINK-7588 URL: https://issues.apache.org/jira/browse/FLINK-7588 Project: Flink Issue Type: Improvement Reporter: Ted Yu In docs/ops/state/large_state_tuning.md , it was mentioned that: bq. the default configuration is tailored towards SSDs and performs suboptimal on spinning disks We should add recommendation targeting spinning disks: https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#difference-of-spinning-disk -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7631) Ineffective check in PageRank#open()
Ted Yu created FLINK-7631: - Summary: Ineffective check in PageRank#open() Key: FLINK-7631 URL: https://issues.apache.org/jira/browse/FLINK-7631 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor >From >flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/library/linkanalysis/PageRank.java > : {code} this.vertexCount = vertexCountIterator.hasNext() ? vertexCountIterator.next().getValue() : 0; this.uniformlyDistributedScore = ((1 - dampingFactor) + dampingFactor * sumOfSinks) / this.vertexCount; {code} The check for vertexCountIterator.hasNext() should enclose the assignments to both this.vertexCount and this.uniformlyDistributedScore Otherwise there may be divide by zero error. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7642) Upgrade maven surefire plugin to 2.19.1
Ted Yu created FLINK-7642: - Summary: Upgrade maven surefire plugin to 2.19.1 Key: FLINK-7642 URL: https://issues.apache.org/jira/browse/FLINK-7642 Project: Flink Issue Type: Improvement Reporter: Ted Yu Surefire 2.19 release introduced more useful test filters which would let us run a subset of the test. This issue is for upgrading maven surefire plugin to 2.19.1 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7659) Unprotected access to inProgress in JobCancellationWithSavepointHandlers#handleNewRequest
Ted Yu created FLINK-7659: - Summary: Unprotected access to inProgress in JobCancellationWithSavepointHandlers#handleNewRequest Key: FLINK-7659 URL: https://issues.apache.org/jira/browse/FLINK-7659 Project: Flink Issue Type: Bug Reporter: Ted Yu Here is related code: {code} } finally { inProgress.remove(jobId); } {code} A little lower, in another finally block, there is: {code} synchronized (lock) { if (!success) { inProgress.remove(jobId); {code} which is correct. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7679) Upgrade maven enforcer plugin to 3.0.0-M1
Ted Yu created FLINK-7679: - Summary: Upgrade maven enforcer plugin to 3.0.0-M1 Key: FLINK-7679 URL: https://issues.apache.org/jira/browse/FLINK-7679 Project: Flink Issue Type: Bug Reporter: Ted Yu I got the following build error against Java 9: {code} [ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce (enforce-maven) on project flink-parent: Execution enforce-maven of goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce failed: An API incompatibility was encountered while executing org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce: java.lang.ExceptionInInitializerError: null [ERROR] - [ERROR] realm =plugin>org.apache.maven.plugins:maven-enforcer-plugin:1.4.1 [ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy [ERROR] urls[0] = file:/home/hbase/.m2/repository/org/apache/maven/plugins/maven-enforcer-plugin/1.4.1/maven-enforcer-plugin-1.4.1.jar {code} Upgrading maven enforcer plugin to 3.0.0-M1 would get over the above error. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7775) Remove unreferenced method PermanentBlobCache#getNumberOfCachedJobs
Ted Yu created FLINK-7775: - Summary: Remove unreferenced method PermanentBlobCache#getNumberOfCachedJobs Key: FLINK-7775 URL: https://issues.apache.org/jira/browse/FLINK-7775 Project: Flink Issue Type: Task Reporter: Ted Yu Priority: Minor {code} public int getNumberOfCachedJobs() { return jobRefCounters.size(); } {code} The method is not used. We should remove it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7795) Utilize error-prone to discover common coding mistakes
Ted Yu created FLINK-7795: - Summary: Utilize error-prone to discover common coding mistakes Key: FLINK-7795 URL: https://issues.apache.org/jira/browse/FLINK-7795 Project: Flink Issue Type: Improvement Reporter: Ted Yu http://errorprone.info/ is a tool which detects common coding mistakes. We should incorporate into Flink build. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7796) RocksDBKeyedStateBackend#RocksDBFullSnapshotOperation should close snapshotCloseableRegistry
Ted Yu created FLINK-7796: - Summary: RocksDBKeyedStateBackend#RocksDBFullSnapshotOperation should close snapshotCloseableRegistry Key: FLINK-7796 URL: https://issues.apache.org/jira/browse/FLINK-7796 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor snapshotCloseableRegistry, being CloseableRegistry, depends on invocation of close() method to release certain resource. It seems close() can be called from releaseSnapshotResources() -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7877) Fix compilation against the Hadoop 3 beta1 release
Ted Yu created FLINK-7877: - Summary: Fix compilation against the Hadoop 3 beta1 release Key: FLINK-7877 URL: https://issues.apache.org/jira/browse/FLINK-7877 Project: Flink Issue Type: Bug Reporter: Ted Yu When compiling against hadoop 3.0.0-beta1, I got: {code} [ERROR] /mnt/disk2/a/flink/flink-yarn/src/test/java/org/apache/flink/yarn/UtilsTest.java:[224,16] org.apache.flink.yarn.UtilsTest.TestingContainer is not abstract and does not override abstract method setExecutionType(org.apache.hadoop.yarn.api.records.ExecutionType) in org.apache.hadoop.yarn.api.records.Container {code} There may other hadoop API(s) that need adjustment. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7897) Consider using nio.Files for file deletion in TransientBlobCleanupTask
Ted Yu created FLINK-7897: - Summary: Consider using nio.Files for file deletion in TransientBlobCleanupTask Key: FLINK-7897 URL: https://issues.apache.org/jira/browse/FLINK-7897 Project: Flink Issue Type: Improvement Reporter: Ted Yu Priority: Minor nio.Files#delete() provides better clue as to why the deletion may fail: https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#delete(java.nio.file.Path) Depending on the potential exception, the call to localFile.exists() may be skipped. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7917) The return of taskInformationOrBlobKey should be placed inside synchronized in ExecutionJobVertex
Ted Yu created FLINK-7917: - Summary: The return of taskInformationOrBlobKey should be placed inside synchronized in ExecutionJobVertex Key: FLINK-7917 URL: https://issues.apache.org/jira/browse/FLINK-7917 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor Currently: {code} } return taskInformationOrBlobKey; {code} The return should be placed inside synchronized block. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7936) Lack of synchronization w.r.t. taskManagers in MetricStore#add()
Ted Yu created FLINK-7936: - Summary: Lack of synchronization w.r.t. taskManagers in MetricStore#add() Key: FLINK-7936 URL: https://issues.apache.org/jira/browse/FLINK-7936 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} String tmID = ((QueryScopeInfo.TaskManagerQueryScopeInfo) info).taskManagerID; tm = taskManagers.computeIfAbsent(tmID, k -> new TaskManagerMetricStore()); {code} In other places, access to taskManagers is protected by lock on MetricStore.this -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7966) Create flink-connector-kafka-1.0 submodule
Ted Yu created FLINK-7966: - Summary: Create flink-connector-kafka-1.0 submodule Key: FLINK-7966 URL: https://issues.apache.org/jira/browse/FLINK-7966 Project: Flink Issue Type: Improvement Reporter: Ted Yu Priority: Major Kafka 1.0 has been released. When compiling flink-connector-kafka-0.11 against 1.0.0 release, I got: {code} [ERROR] /a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[468,30] cannot find symbol [ERROR] symbol: method setTransactionTimeout(long) [ERROR] /a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[469,30] cannot find symbol [ERROR] symbol: method enableTransactionTimeoutWarnings(double) [ERROR] /a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[509,9] method does not override or implement a method from a supertype [ERROR] /a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[511,22] cannot find symbol [ERROR] symbol: method ignoreFailuresAfterTransactionTimeout() [ERROR] /a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[586,66] cannot find symbol [ERROR] symbol: method currentTransaction() [ERROR] location: class org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011 [ERROR] /a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[619,82] cannot find symbol [ERROR] symbol: method currentTransaction() [ERROR] location: class org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011 [ERROR] /a/flink/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer011.java:[895,66] cannot find symbol [ERROR] symbol: method currentTransaction() [ERROR] location: class org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011 {code} We should create flink-connector-kafka-1.0 submodule to accommodate the new Kafka release -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8036) Consider using gradle to build Flink
Ted Yu created FLINK-8036: - Summary: Consider using gradle to build Flink Key: FLINK-8036 URL: https://issues.apache.org/jira/browse/FLINK-8036 Project: Flink Issue Type: Improvement Reporter: Ted Yu Here is summary from Lukasz over this thread (http://search-hadoop.com/m/Beam/gfKHFVh4NM151XIu1?subj=Re+DISCUSS+Move+away+from+Apache+Maven+as+build+tool) w.r.t. performance boost from using gradle: Maven performs parallelization at the module level, an entire module needs to complete before any dependent modules can start, this means running all the checks like findbugs, checkstyle, tests need to finish. Gradle has task level parallelism between subprojects which means that as soon as the compile and shade steps are done for a project, and dependent subprojects can typically start. This means that we get increased parallelism due to not needing to wait for findbugs, checkstyle, tests to run. I typically see ~20 tasks (at peak) running on my desktop in parallel. Flink should consider using gradle - on Linux with SSD, a clean build takes an hour. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8037) Missing cast in integer arithmetic in TransactionalIdsGenerator#generateIdsToAbort
Ted Yu created FLINK-8037: - Summary: Missing cast in integer arithmetic in TransactionalIdsGenerator#generateIdsToAbort Key: FLINK-8037 URL: https://issues.apache.org/jira/browse/FLINK-8037 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} public Set generateIdsToAbort() { Set idsToAbort = new HashSet<>(); for (int i = 0; i < safeScaleDownFactor; i++) { idsToAbort.addAll(generateIdsToUse(i * poolSize * totalNumberOfSubtasks)); {code} The operands are integers where generateIdsToUse() expects long parameter. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8048) RAT check complaint
Ted Yu created FLINK-8048: - Summary: RAT check complaint Key: FLINK-8048 URL: https://issues.apache.org/jira/browse/FLINK-8048 Project: Flink Issue Type: Bug Reporter: Ted Yu Running {{mvn rat:check}} gives warning about the following files: test-infra/end-to-end-test/test-data/words .editorconfig .github/PULL_REQUEST_TEMPLATE.md .github/CONTRIBUTING.md -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8075) Redundant clone() in Configuration ctor
Ted Yu created FLINK-8075: - Summary: Redundant clone() in Configuration ctor Key: FLINK-8075 URL: https://issues.apache.org/jira/browse/FLINK-8075 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor In flink-filesystems/flink-s3-fs-presto/src/main/java/org/apache/hadoop/conf/Configuration.java , at line 703: {code} public Configuration(Configuration other) { this.resources = (ArrayList) other.resources.clone(); synchronized(other) { if (other.properties != null) { this.properties = (Properties)other.properties.clone(); } {code} The first clone() call is without synchronization and without null check. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8100) Consider introducing log4j-extras
Ted Yu created FLINK-8100: - Summary: Consider introducing log4j-extras Key: FLINK-8100 URL: https://issues.apache.org/jira/browse/FLINK-8100 Project: Flink Issue Type: Improvement Reporter: Ted Yu log4j-extras allows log rotation as well as compression. https://logging.apache.org/log4j/extras/download.html We should consider using log4j-extras. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8146) Potential resource leak in PythonPlanBinder#unzipPythonLibrary
Ted Yu created FLINK-8146: - Summary: Potential resource leak in PythonPlanBinder#unzipPythonLibrary Key: FLINK-8146 URL: https://issues.apache.org/jira/browse/FLINK-8146 Project: Flink Issue Type: Bug Reporter: Ted Yu {code} while (entry != null) { ... } zis.closeEntry(); {code} Looking at the catch block inside the loop, it seems the intention is to close zis upon getting exception. zis.close() should be called outside the loop. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8198) Useless check against -1 in LimitedConnectionsFileSystem#fromConfig
Ted Yu created FLINK-8198: - Summary: Useless check against -1 in LimitedConnectionsFileSystem#fromConfig Key: FLINK-8198 URL: https://issues.apache.org/jira/browse/FLINK-8198 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} return new ConnectionLimitingSettings( totalLimit == -1 ? 0 : totalLimit, limitIn == -1 ? 0 : limitIn, limitOut == -1 ? 0 : limitOut, openTimeout, {code} If any of the 3 variables is negative, control would have returned in the if block above. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8335) Upgrade hbase connector dependency to 1.4.0
Ted Yu created FLINK-8335: - Summary: Upgrade hbase connector dependency to 1.4.0 Key: FLINK-8335 URL: https://issues.apache.org/jira/browse/FLINK-8335 Project: Flink Issue Type: Improvement Reporter: Ted Yu Priority: Minor hbase 1.4.0 has been released. 1.4.0 shows speed improvement over previous 1.x releases. This issue is to upgrade the dependency to 1.4.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8394) Lack of synchronization accessing expectedRecord in ReceiverThread#shutdown
Ted Yu created FLINK-8394: - Summary: Lack of synchronization accessing expectedRecord in ReceiverThread#shutdown Key: FLINK-8394 URL: https://issues.apache.org/jira/browse/FLINK-8394 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} public void shutdown() { running = false; interrupt(); expectedRecord.complete(0L); {code} Access to expectedRecord should be protected by synchronization, as done on other methods. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8415) Unprotected access to recordsToSend in LongRecordWriterThread#shutdown()
Ted Yu created FLINK-8415: - Summary: Unprotected access to recordsToSend in LongRecordWriterThread#shutdown() Key: FLINK-8415 URL: https://issues.apache.org/jira/browse/FLINK-8415 Project: Flink Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} public void shutdown() { running = false; recordsToSend.complete(0L); {code} In other methods, access to recordsToSend is protected by synchronized keyword. shutdown() should do the same. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-8464) TestRaftReconfigurationWithSimulatedRpc fails with assertion error
Ted Yu created FLINK-8464: - Summary: TestRaftReconfigurationWithSimulatedRpc fails with assertion error Key: FLINK-8464 URL: https://issues.apache.org/jira/browse/FLINK-8464 Project: Flink Issue Type: Test Reporter: Ted Yu Attachments: ratis-8464.tst As of commit 7b3a9a6f5f8e8075727d84e3ddeae7b594eda89c, I observed the following : {code} testRevertConfigurationChange(org.apache.ratis.server.simulation.TestRaftReconfigurationWithSimulatedRpc) Time elapsed: 2.119 sec <<< FAILURE! java.lang.AssertionError: 1 0 expected: but was: {code} 1 was confIndex and 0 was log.getLastCommittedIndex() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8554) Upgrade AWS SDK
Ted Yu created FLINK-8554: - Summary: Upgrade AWS SDK Key: FLINK-8554 URL: https://issues.apache.org/jira/browse/FLINK-8554 Project: Flink Issue Type: Improvement Reporter: Ted Yu AWS SDK 1.11.271 fixes a lot of bugs. One of which would exhibit the following: {code} Caused by: java.lang.NullPointerException at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729) at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67) at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8661) Replace Collections.EMPTY_MAP with Collections.emptyMap()
Ted Yu created FLINK-8661: - Summary: Replace Collections.EMPTY_MAP with Collections.emptyMap() Key: FLINK-8661 URL: https://issues.apache.org/jira/browse/FLINK-8661 Project: Flink Issue Type: Bug Reporter: Ted Yu The use of Collections.EMPTY_SET and Collections.EMPTY_MAP often causes unchecked assignment and it should be replaced with Collections.emptySet() and Collections.emptyMap() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-8687) MaterializedCollectStreamResult#retrievePage should take resultLock
Ted Yu created FLINK-8687: - Summary: MaterializedCollectStreamResult#retrievePage should take resultLock Key: FLINK-8687 URL: https://issues.apache.org/jira/browse/FLINK-8687 Project: Flink Issue Type: Bug Reporter: Ted Yu Currently MaterializedCollectStreamResult#retrievePage checks page range and calls snapshot.subList() without holding resultLock. resultLock should be taken. -- This message was sent by Atlassian JIRA (v7.6.3#76005)