[jira] [Created] (FLINK-10730) How to resume Flink SQL Job ?
xinzhang created FLINK-10730: Summary: How to resume Flink SQL Job ? Key: FLINK-10730 URL: https://issues.apache.org/jira/browse/FLINK-10730 Project: Flink Issue Type: Improvement Components: Job-Submission, SQL Client, YARN Affects Versions: 1.6.2 Reporter: xinzhang Hi. Is any way to resume Flink SQL Job ? The guide here [flink title|https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/savepoints.html#resuming-from-savepoints] means flink resume Flink Job (Like some Jar) but no Flink SQL Job. 1.Now I submit the Flink SQL Job use the cmd： cat test.sql |xargs -0 $FLINK_HOME/bin/sql-client.sh embedded -e sql-client-kafka-json.yaml --library $FLINK_HOME/lib -u -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10730) How to resume Flink SQL Job ?
[ https://issues.apache.org/jira/browse/FLINK-10730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xinzhang updated FLINK-10730: - Component/s: (was: YARN) (was: Job-Submission) > How to resume Flink SQL Job ? > - > > Key: FLINK-10730 > URL: https://issues.apache.org/jira/browse/FLINK-10730 > Project: Flink > Issue Type: Improvement > Components: SQL Client >Affects Versions: 1.6.2 >Reporter: xinzhang >Priority: Major > > Hi. Is any way to resume Flink SQL Job ? The guide here [flink > title|https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/savepoints.html#resuming-from-savepoints] > means flink resume Flink Job (Like some Jar) but no Flink SQL Job. > 1.Now I submit the Flink SQL Job use the cmd： > cat test.sql |xargs -0 $FLINK_HOME/bin/sql-client.sh embedded -e > sql-client-kafka-json.yaml --library $FLINK_HOME/lib -u > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] twalthr commented on a change in pull request #6966: [FLINK-10166] [table] Reduce dependencies by removing org.apache.commons
twalthr commented on a change in pull request #6966: [FLINK-10166] [table] Reduce dependencies by removing org.apache.commons URL: https://github.com/apache/flink/pull/6966#discussion_r229598889 ## File path: flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java ## @@ -40,14 +44,18 @@ private static final Base64.Decoder BASE64_DECODER = java.util.Base64.getUrlDecoder(); + private static final MessageDigest MD5_MESSAGE_DIGEST = getMd5MessageDigest(); Review comment: Usually this doesn't matter for translation (which is single threaded) but we are on the safer side if this method is thread-safe in the future. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10166) Dependency problems when executing SQL query in sql-client
[
https://issues.apache.org/jira/browse/FLINK-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669738#comment-16669738
]
ASF GitHub Bot commented on FLINK-10166:
twalthr commented on a change in pull request #6966: [FLINK-10166] [table]
Reduce dependencies by removing org.apache.commons
URL: https://github.com/apache/flink/pull/6966#discussion_r229598889
##
File path:
flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
##
@@ -40,14 +44,18 @@
private static final Base64.Decoder BASE64_DECODER =
java.util.Base64.getUrlDecoder();
+ private static final MessageDigest MD5_MESSAGE_DIGEST =
getMd5MessageDigest();
Review comment:
Usually this doesn't matter for translation (which is single threaded) but
we are on the safer side if this method is thread-safe in the future.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Dependency problems when executing SQL query in sql-client
> --
>
> Key: FLINK-10166
> URL: https://issues.apache.org/jira/browse/FLINK-10166
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.6.0
>Reporter: Dawid Wysakowicz
>Assignee: Timo Walther
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> When tried to run query:
> {code}
> select count(distinct name) from (Values ('a'), ('b')) AS NameTable(name)
> {code}
> in {{sql-client.sh}} I got:
> {code}
> [ERROR] Could not execute SQL statement. Reason:
> org.codehaus.commons.compiler.CompileException: Line 43, Column 10: Unknown
> variable or type "org.apache.commons.codec.binary.Base64"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] twalthr commented on a change in pull request #6966: [FLINK-10166] [table] Reduce dependencies by removing org.apache.commons
twalthr commented on a change in pull request #6966: [FLINK-10166] [table]
Reduce dependencies by removing org.apache.commons
URL: https://github.com/apache/flink/pull/6966#discussion_r229599636
##
File path:
flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
##
@@ -87,10 +95,145 @@ public static String encodeObjectToString(Serializable
obj) {
return loadClass(qualifiedName,
Thread.currentThread().getContextClassLoader());
}
+ public static String encodeStringToBase64(String string) {
+ return new
String(java.util.Base64.getEncoder().encode(string.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static String decodeBase64ToString(String base64) {
+ return new
String(java.util.Base64.getDecoder().decode(base64.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static byte[] md5(String string) {
+ if (MD5_MESSAGE_DIGEST == null) {
+ throw new TableException("Unsupported MD5 algorithm.");
+ }
+ return MD5_MESSAGE_DIGEST.digest(string.getBytes(UTF_8));
+ }
+
+ public static String hex(String string) {
+ return hex(string.getBytes(UTF_8));
+ }
+
+ public static String hex(byte[] bytes) {
+ // adopted from https://stackoverflow.com/a/9855338
+ final char[] hexChars = new char[bytes.length * 2];
+ for (int j = 0; j < bytes.length; j++) {
+ final int v = bytes[j] & 0xFF;
+ hexChars[j * 2] = HEX_CHARS[v >>> 4];
+ hexChars[j * 2 + 1] = HEX_CHARS[v & 0x0F];
+ }
+ return new String(hexChars);
+ }
+
+ private static MessageDigest getMd5MessageDigest() {
+ try {
+ return MessageDigest.getInstance("MD5");
+ } catch (NoSuchAlgorithmException e) {
+ return null;
+ }
+ }
+
+ //
+ // Java String Repetition
+ //
+ // copied from o.a.commons.lang3.StringUtils (commons-lang3:3.3.2)
+ //
+
+ private static final String EMPTY = "";
+
+ /**
+* The maximum size to which the padding constant(s) can expand.
+*/
+ private static final int PAD_LIMIT = 8192;
+
+ /**
+* Repeat a String {@code repeat} times to form a new String.
+*
+*
+* StringUtils.repeat(null, 2) = null
+* StringUtils.repeat("", 0) = ""
+* StringUtils.repeat("", 2) = ""
+* StringUtils.repeat("a", 3) = "aaa"
+* StringUtils.repeat("ab", 2) = "abab"
+* StringUtils.repeat("a", -2) = ""
+*
+*
+* @param strthe String to repeat, may be null
+* @param repeat number of times to repeat str, negative treated as zero
+* @return a new String consisting of the original String repeated,
{@code null} if null String input
+*/
+ public static String repeat(final String str, final int repeat) {
+ // Performance tuned for 2.0 (JDK1.4)
+
+ if (str == null) {
+ return null;
+ }
+ if (repeat <= 0) {
+ return EMPTY;
+ }
+ final int inputLength = str.length();
+ if (repeat == 1 || inputLength == 0) {
+ return str;
+ }
+ if (inputLength == 1 && repeat <= PAD_LIMIT) {
+ return repeat(str.charAt(0), repeat);
+ }
+
+ final int outputLength = inputLength * repeat;
+ switch (inputLength) {
+ case 1:
+ return repeat(str.charAt(0), repeat);
+ case 2:
+ final char ch0 = str.charAt(0);
+ final char ch1 = str.charAt(1);
+ final char[] output2 = new char[outputLength];
+ for (int i = repeat * 2 - 2; i >= 0; i--, i--) {
+ output2[i] = ch0;
+ output2[i + 1] = ch1;
+ }
+ return new String(output2);
+ default:
+ final StringBuilder buf = new
StringBuilder(outputLength);
Review comment:
If I change this method, I would also need to introduce more extensive tests
in order to verify that I did not introduce new bugs. This method can be remove
soon anyway as Java 11 has a `String.repeat` method.
---
[jira] [Commented] (FLINK-10166) Dependency problems when executing SQL query in sql-client
[
https://issues.apache.org/jira/browse/FLINK-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669740#comment-16669740
]
ASF GitHub Bot commented on FLINK-10166:
twalthr commented on a change in pull request #6966: [FLINK-10166] [table]
Reduce dependencies by removing org.apache.commons
URL: https://github.com/apache/flink/pull/6966#discussion_r229599636
##
File path:
flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
##
@@ -87,10 +95,145 @@ public static String encodeObjectToString(Serializable
obj) {
return loadClass(qualifiedName,
Thread.currentThread().getContextClassLoader());
}
+ public static String encodeStringToBase64(String string) {
+ return new
String(java.util.Base64.getEncoder().encode(string.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static String decodeBase64ToString(String base64) {
+ return new
String(java.util.Base64.getDecoder().decode(base64.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static byte[] md5(String string) {
+ if (MD5_MESSAGE_DIGEST == null) {
+ throw new TableException("Unsupported MD5 algorithm.");
+ }
+ return MD5_MESSAGE_DIGEST.digest(string.getBytes(UTF_8));
+ }
+
+ public static String hex(String string) {
+ return hex(string.getBytes(UTF_8));
+ }
+
+ public static String hex(byte[] bytes) {
+ // adopted from https://stackoverflow.com/a/9855338
+ final char[] hexChars = new char[bytes.length * 2];
+ for (int j = 0; j < bytes.length; j++) {
+ final int v = bytes[j] & 0xFF;
+ hexChars[j * 2] = HEX_CHARS[v >>> 4];
+ hexChars[j * 2 + 1] = HEX_CHARS[v & 0x0F];
+ }
+ return new String(hexChars);
+ }
+
+ private static MessageDigest getMd5MessageDigest() {
+ try {
+ return MessageDigest.getInstance("MD5");
+ } catch (NoSuchAlgorithmException e) {
+ return null;
+ }
+ }
+
+ //
+ // Java String Repetition
+ //
+ // copied from o.a.commons.lang3.StringUtils (commons-lang3:3.3.2)
+ //
+
+ private static final String EMPTY = "";
+
+ /**
+* The maximum size to which the padding constant(s) can expand.
+*/
+ private static final int PAD_LIMIT = 8192;
+
+ /**
+* Repeat a String {@code repeat} times to form a new String.
+*
+*
+* StringUtils.repeat(null, 2) = null
+* StringUtils.repeat("", 0) = ""
+* StringUtils.repeat("", 2) = ""
+* StringUtils.repeat("a", 3) = "aaa"
+* StringUtils.repeat("ab", 2) = "abab"
+* StringUtils.repeat("a", -2) = ""
+*
+*
+* @param strthe String to repeat, may be null
+* @param repeat number of times to repeat str, negative treated as zero
+* @return a new String consisting of the original String repeated,
{@code null} if null String input
+*/
+ public static String repeat(final String str, final int repeat) {
+ // Performance tuned for 2.0 (JDK1.4)
+
+ if (str == null) {
+ return null;
+ }
+ if (repeat <= 0) {
+ return EMPTY;
+ }
+ final int inputLength = str.length();
+ if (repeat == 1 || inputLength == 0) {
+ return str;
+ }
+ if (inputLength == 1 && repeat <= PAD_LIMIT) {
+ return repeat(str.charAt(0), repeat);
+ }
+
+ final int outputLength = inputLength * repeat;
+ switch (inputLength) {
+ case 1:
+ return repeat(str.charAt(0), repeat);
+ case 2:
+ final char ch0 = str.charAt(0);
+ final char ch1 = str.charAt(1);
+ final char[] output2 = new char[outputLength];
+ for (int i = repeat * 2 - 2; i >= 0; i--, i--) {
+ output2[i] = ch0;
+ output2[i + 1] = ch1;
+ }
+ return new String(output2);
+ default:
+ final StringBuilder buf = new
StringBuilder(outputLen
[GitHub] twalthr commented on a change in pull request #6966: [FLINK-10166] [table] Reduce dependencies by removing org.apache.commons
twalthr commented on a change in pull request #6966: [FLINK-10166] [table]
Reduce dependencies by removing org.apache.commons
URL: https://github.com/apache/flink/pull/6966#discussion_r229599636
##
File path:
flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
##
@@ -87,10 +95,145 @@ public static String encodeObjectToString(Serializable
obj) {
return loadClass(qualifiedName,
Thread.currentThread().getContextClassLoader());
}
+ public static String encodeStringToBase64(String string) {
+ return new
String(java.util.Base64.getEncoder().encode(string.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static String decodeBase64ToString(String base64) {
+ return new
String(java.util.Base64.getDecoder().decode(base64.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static byte[] md5(String string) {
+ if (MD5_MESSAGE_DIGEST == null) {
+ throw new TableException("Unsupported MD5 algorithm.");
+ }
+ return MD5_MESSAGE_DIGEST.digest(string.getBytes(UTF_8));
+ }
+
+ public static String hex(String string) {
+ return hex(string.getBytes(UTF_8));
+ }
+
+ public static String hex(byte[] bytes) {
+ // adopted from https://stackoverflow.com/a/9855338
+ final char[] hexChars = new char[bytes.length * 2];
+ for (int j = 0; j < bytes.length; j++) {
+ final int v = bytes[j] & 0xFF;
+ hexChars[j * 2] = HEX_CHARS[v >>> 4];
+ hexChars[j * 2 + 1] = HEX_CHARS[v & 0x0F];
+ }
+ return new String(hexChars);
+ }
+
+ private static MessageDigest getMd5MessageDigest() {
+ try {
+ return MessageDigest.getInstance("MD5");
+ } catch (NoSuchAlgorithmException e) {
+ return null;
+ }
+ }
+
+ //
+ // Java String Repetition
+ //
+ // copied from o.a.commons.lang3.StringUtils (commons-lang3:3.3.2)
+ //
+
+ private static final String EMPTY = "";
+
+ /**
+* The maximum size to which the padding constant(s) can expand.
+*/
+ private static final int PAD_LIMIT = 8192;
+
+ /**
+* Repeat a String {@code repeat} times to form a new String.
+*
+*
+* StringUtils.repeat(null, 2) = null
+* StringUtils.repeat("", 0) = ""
+* StringUtils.repeat("", 2) = ""
+* StringUtils.repeat("a", 3) = "aaa"
+* StringUtils.repeat("ab", 2) = "abab"
+* StringUtils.repeat("a", -2) = ""
+*
+*
+* @param strthe String to repeat, may be null
+* @param repeat number of times to repeat str, negative treated as zero
+* @return a new String consisting of the original String repeated,
{@code null} if null String input
+*/
+ public static String repeat(final String str, final int repeat) {
+ // Performance tuned for 2.0 (JDK1.4)
+
+ if (str == null) {
+ return null;
+ }
+ if (repeat <= 0) {
+ return EMPTY;
+ }
+ final int inputLength = str.length();
+ if (repeat == 1 || inputLength == 0) {
+ return str;
+ }
+ if (inputLength == 1 && repeat <= PAD_LIMIT) {
+ return repeat(str.charAt(0), repeat);
+ }
+
+ final int outputLength = inputLength * repeat;
+ switch (inputLength) {
+ case 1:
+ return repeat(str.charAt(0), repeat);
+ case 2:
+ final char ch0 = str.charAt(0);
+ final char ch1 = str.charAt(1);
+ final char[] output2 = new char[outputLength];
+ for (int i = repeat * 2 - 2; i >= 0; i--, i--) {
+ output2[i] = ch0;
+ output2[i + 1] = ch1;
+ }
+ return new String(output2);
+ default:
+ final StringBuilder buf = new
StringBuilder(outputLength);
Review comment:
If I change this method, I would also need to introduce more extensive tests
in order to verify that I did not introduce new bugs. This method can be
removed soon anyway as Java 11 has a `String.repeat` method.
--
[jira] [Commented] (FLINK-10166) Dependency problems when executing SQL query in sql-client
[
https://issues.apache.org/jira/browse/FLINK-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669741#comment-16669741
]
ASF GitHub Bot commented on FLINK-10166:
twalthr commented on a change in pull request #6966: [FLINK-10166] [table]
Reduce dependencies by removing org.apache.commons
URL: https://github.com/apache/flink/pull/6966#discussion_r229599636
##
File path:
flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
##
@@ -87,10 +95,145 @@ public static String encodeObjectToString(Serializable
obj) {
return loadClass(qualifiedName,
Thread.currentThread().getContextClassLoader());
}
+ public static String encodeStringToBase64(String string) {
+ return new
String(java.util.Base64.getEncoder().encode(string.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static String decodeBase64ToString(String base64) {
+ return new
String(java.util.Base64.getDecoder().decode(base64.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static byte[] md5(String string) {
+ if (MD5_MESSAGE_DIGEST == null) {
+ throw new TableException("Unsupported MD5 algorithm.");
+ }
+ return MD5_MESSAGE_DIGEST.digest(string.getBytes(UTF_8));
+ }
+
+ public static String hex(String string) {
+ return hex(string.getBytes(UTF_8));
+ }
+
+ public static String hex(byte[] bytes) {
+ // adopted from https://stackoverflow.com/a/9855338
+ final char[] hexChars = new char[bytes.length * 2];
+ for (int j = 0; j < bytes.length; j++) {
+ final int v = bytes[j] & 0xFF;
+ hexChars[j * 2] = HEX_CHARS[v >>> 4];
+ hexChars[j * 2 + 1] = HEX_CHARS[v & 0x0F];
+ }
+ return new String(hexChars);
+ }
+
+ private static MessageDigest getMd5MessageDigest() {
+ try {
+ return MessageDigest.getInstance("MD5");
+ } catch (NoSuchAlgorithmException e) {
+ return null;
+ }
+ }
+
+ //
+ // Java String Repetition
+ //
+ // copied from o.a.commons.lang3.StringUtils (commons-lang3:3.3.2)
+ //
+
+ private static final String EMPTY = "";
+
+ /**
+* The maximum size to which the padding constant(s) can expand.
+*/
+ private static final int PAD_LIMIT = 8192;
+
+ /**
+* Repeat a String {@code repeat} times to form a new String.
+*
+*
+* StringUtils.repeat(null, 2) = null
+* StringUtils.repeat("", 0) = ""
+* StringUtils.repeat("", 2) = ""
+* StringUtils.repeat("a", 3) = "aaa"
+* StringUtils.repeat("ab", 2) = "abab"
+* StringUtils.repeat("a", -2) = ""
+*
+*
+* @param strthe String to repeat, may be null
+* @param repeat number of times to repeat str, negative treated as zero
+* @return a new String consisting of the original String repeated,
{@code null} if null String input
+*/
+ public static String repeat(final String str, final int repeat) {
+ // Performance tuned for 2.0 (JDK1.4)
+
+ if (str == null) {
+ return null;
+ }
+ if (repeat <= 0) {
+ return EMPTY;
+ }
+ final int inputLength = str.length();
+ if (repeat == 1 || inputLength == 0) {
+ return str;
+ }
+ if (inputLength == 1 && repeat <= PAD_LIMIT) {
+ return repeat(str.charAt(0), repeat);
+ }
+
+ final int outputLength = inputLength * repeat;
+ switch (inputLength) {
+ case 1:
+ return repeat(str.charAt(0), repeat);
+ case 2:
+ final char ch0 = str.charAt(0);
+ final char ch1 = str.charAt(1);
+ final char[] output2 = new char[outputLength];
+ for (int i = repeat * 2 - 2; i >= 0; i--, i--) {
+ output2[i] = ch0;
+ output2[i + 1] = ch1;
+ }
+ return new String(output2);
+ default:
+ final StringBuilder buf = new
StringBuilder(outputLen
[jira] [Commented] (FLINK-7286) Flink Dashboard fails to display bytes/records received by sources
[ https://issues.apache.org/jira/browse/FLINK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669756#comment-16669756 ] lining commented on FLINK-7286: --- Hi, [~aljoscha]. I am doing this now, can assign this to me. > Flink Dashboard fails to display bytes/records received by sources > -- > > Key: FLINK-7286 > URL: https://issues.apache.org/jira/browse/FLINK-7286 > Project: Flink > Issue Type: Improvement > Components: Metrics, Webfrontend >Affects Versions: 1.3.1 >Reporter: Elias Levy >Assignee: Hai Zhou >Priority: Major > Fix For: 1.7.0 > > > It appears Flink can't measure the number of bytes read or records produced > by a source (e.g. Kafka source). This is particularly problematic for simple > jobs where the job pipeline is chained, and in which there are no > measurements between operators. Thus, in the UI it appears that the job is > not consuming any data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-7286) Flink Dashboard fails to display bytes/records received by sources
[ https://issues.apache.org/jira/browse/FLINK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669763#comment-16669763 ] Fabian Wollert commented on FLINK-7286: --- Hi [~lining] , please mind FLINK-9050 which blocks this ticket. it needs to be done first. > Flink Dashboard fails to display bytes/records received by sources > -- > > Key: FLINK-7286 > URL: https://issues.apache.org/jira/browse/FLINK-7286 > Project: Flink > Issue Type: Improvement > Components: Metrics, Webfrontend >Affects Versions: 1.3.1 >Reporter: Elias Levy >Assignee: Hai Zhou >Priority: Major > Fix For: 1.7.0 > > > It appears Flink can't measure the number of bytes read or records produced > by a source (e.g. Kafka source). This is particularly problematic for simple > jobs where the job pipeline is chained, and in which there are no > measurements between operators. Thus, in the UI it appears that the job is > not consuming any data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10731) Support AVG on Date fields
Flavio Pompermaier created FLINK-10731: -- Summary: Support AVG on Date fields Key: FLINK-10731 URL: https://issues.apache.org/jira/browse/FLINK-10731 Project: Flink Issue Type: Improvement Components: Table API & SQL Affects Versions: 1.6.2 Reporter: Flavio Pompermaier AVG function does not work on date fields right now -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] igalshilman commented on a change in pull request #6966: [FLINK-10166] [table] Reduce dependencies by removing org.apache.commons
igalshilman commented on a change in pull request #6966: [FLINK-10166] [table]
Reduce dependencies by removing org.apache.commons
URL: https://github.com/apache/flink/pull/6966#discussion_r229603056
##
File path:
flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
##
@@ -87,10 +95,145 @@ public static String encodeObjectToString(Serializable
obj) {
return loadClass(qualifiedName,
Thread.currentThread().getContextClassLoader());
}
+ public static String encodeStringToBase64(String string) {
+ return new
String(java.util.Base64.getEncoder().encode(string.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static String decodeBase64ToString(String base64) {
+ return new
String(java.util.Base64.getDecoder().decode(base64.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static byte[] md5(String string) {
+ if (MD5_MESSAGE_DIGEST == null) {
+ throw new TableException("Unsupported MD5 algorithm.");
+ }
+ return MD5_MESSAGE_DIGEST.digest(string.getBytes(UTF_8));
+ }
+
+ public static String hex(String string) {
+ return hex(string.getBytes(UTF_8));
+ }
+
+ public static String hex(byte[] bytes) {
+ // adopted from https://stackoverflow.com/a/9855338
+ final char[] hexChars = new char[bytes.length * 2];
+ for (int j = 0; j < bytes.length; j++) {
+ final int v = bytes[j] & 0xFF;
+ hexChars[j * 2] = HEX_CHARS[v >>> 4];
+ hexChars[j * 2 + 1] = HEX_CHARS[v & 0x0F];
+ }
+ return new String(hexChars);
+ }
+
+ private static MessageDigest getMd5MessageDigest() {
+ try {
+ return MessageDigest.getInstance("MD5");
+ } catch (NoSuchAlgorithmException e) {
+ return null;
+ }
+ }
+
+ //
+ // Java String Repetition
+ //
+ // copied from o.a.commons.lang3.StringUtils (commons-lang3:3.3.2)
+ //
+
+ private static final String EMPTY = "";
+
+ /**
+* The maximum size to which the padding constant(s) can expand.
+*/
+ private static final int PAD_LIMIT = 8192;
+
+ /**
+* Repeat a String {@code repeat} times to form a new String.
+*
+*
+* StringUtils.repeat(null, 2) = null
+* StringUtils.repeat("", 0) = ""
+* StringUtils.repeat("", 2) = ""
+* StringUtils.repeat("a", 3) = "aaa"
+* StringUtils.repeat("ab", 2) = "abab"
+* StringUtils.repeat("a", -2) = ""
+*
+*
+* @param strthe String to repeat, may be null
+* @param repeat number of times to repeat str, negative treated as zero
+* @return a new String consisting of the original String repeated,
{@code null} if null String input
+*/
+ public static String repeat(final String str, final int repeat) {
+ // Performance tuned for 2.0 (JDK1.4)
+
+ if (str == null) {
+ return null;
+ }
+ if (repeat <= 0) {
+ return EMPTY;
+ }
+ final int inputLength = str.length();
+ if (repeat == 1 || inputLength == 0) {
+ return str;
+ }
+ if (inputLength == 1 && repeat <= PAD_LIMIT) {
+ return repeat(str.charAt(0), repeat);
+ }
+
+ final int outputLength = inputLength * repeat;
+ switch (inputLength) {
+ case 1:
+ return repeat(str.charAt(0), repeat);
+ case 2:
+ final char ch0 = str.charAt(0);
+ final char ch1 = str.charAt(1);
+ final char[] output2 = new char[outputLength];
+ for (int i = repeat * 2 - 2; i >= 0; i--, i--) {
+ output2[i] = ch0;
+ output2[i + 1] = ch1;
+ }
+ return new String(output2);
+ default:
+ final StringBuilder buf = new
StringBuilder(outputLength);
Review comment:
Got it. makes senes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the
[GitHub] igalshilman commented on issue #6966: [FLINK-10166] [table] Reduce dependencies by removing org.apache.commons
igalshilman commented on issue #6966: [FLINK-10166] [table] Reduce dependencies by removing org.apache.commons URL: https://github.com/apache/flink/pull/6966#issuecomment-434604624 LGTM 👍 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] igalshilman commented on a change in pull request #6966: [FLINK-10166] [table] Reduce dependencies by removing org.apache.commons
igalshilman commented on a change in pull request #6966: [FLINK-10166] [table] Reduce dependencies by removing org.apache.commons URL: https://github.com/apache/flink/pull/6966#discussion_r229603180 ## File path: flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java ## @@ -40,14 +44,18 @@ private static final Base64.Decoder BASE64_DECODER = java.util.Base64.getUrlDecoder(); + private static final MessageDigest MD5_MESSAGE_DIGEST = getMd5MessageDigest(); Review comment: Thanks for addressing this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10166) Dependency problems when executing SQL query in sql-client
[
https://issues.apache.org/jira/browse/FLINK-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669765#comment-16669765
]
ASF GitHub Bot commented on FLINK-10166:
igalshilman commented on a change in pull request #6966: [FLINK-10166] [table]
Reduce dependencies by removing org.apache.commons
URL: https://github.com/apache/flink/pull/6966#discussion_r229603180
##
File path:
flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
##
@@ -40,14 +44,18 @@
private static final Base64.Decoder BASE64_DECODER =
java.util.Base64.getUrlDecoder();
+ private static final MessageDigest MD5_MESSAGE_DIGEST =
getMd5MessageDigest();
Review comment:
Thanks for addressing this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Dependency problems when executing SQL query in sql-client
> --
>
> Key: FLINK-10166
> URL: https://issues.apache.org/jira/browse/FLINK-10166
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.6.0
>Reporter: Dawid Wysakowicz
>Assignee: Timo Walther
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> When tried to run query:
> {code}
> select count(distinct name) from (Values ('a'), ('b')) AS NameTable(name)
> {code}
> in {{sql-client.sh}} I got:
> {code}
> [ERROR] Could not execute SQL statement. Reason:
> org.codehaus.commons.compiler.CompileException: Line 43, Column 10: Unknown
> variable or type "org.apache.commons.codec.binary.Base64"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10166) Dependency problems when executing SQL query in sql-client
[
https://issues.apache.org/jira/browse/FLINK-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669764#comment-16669764
]
ASF GitHub Bot commented on FLINK-10166:
igalshilman commented on a change in pull request #6966: [FLINK-10166] [table]
Reduce dependencies by removing org.apache.commons
URL: https://github.com/apache/flink/pull/6966#discussion_r229603056
##
File path:
flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
##
@@ -87,10 +95,145 @@ public static String encodeObjectToString(Serializable
obj) {
return loadClass(qualifiedName,
Thread.currentThread().getContextClassLoader());
}
+ public static String encodeStringToBase64(String string) {
+ return new
String(java.util.Base64.getEncoder().encode(string.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static String decodeBase64ToString(String base64) {
+ return new
String(java.util.Base64.getDecoder().decode(base64.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static byte[] md5(String string) {
+ if (MD5_MESSAGE_DIGEST == null) {
+ throw new TableException("Unsupported MD5 algorithm.");
+ }
+ return MD5_MESSAGE_DIGEST.digest(string.getBytes(UTF_8));
+ }
+
+ public static String hex(String string) {
+ return hex(string.getBytes(UTF_8));
+ }
+
+ public static String hex(byte[] bytes) {
+ // adopted from https://stackoverflow.com/a/9855338
+ final char[] hexChars = new char[bytes.length * 2];
+ for (int j = 0; j < bytes.length; j++) {
+ final int v = bytes[j] & 0xFF;
+ hexChars[j * 2] = HEX_CHARS[v >>> 4];
+ hexChars[j * 2 + 1] = HEX_CHARS[v & 0x0F];
+ }
+ return new String(hexChars);
+ }
+
+ private static MessageDigest getMd5MessageDigest() {
+ try {
+ return MessageDigest.getInstance("MD5");
+ } catch (NoSuchAlgorithmException e) {
+ return null;
+ }
+ }
+
+ //
+ // Java String Repetition
+ //
+ // copied from o.a.commons.lang3.StringUtils (commons-lang3:3.3.2)
+ //
+
+ private static final String EMPTY = "";
+
+ /**
+* The maximum size to which the padding constant(s) can expand.
+*/
+ private static final int PAD_LIMIT = 8192;
+
+ /**
+* Repeat a String {@code repeat} times to form a new String.
+*
+*
+* StringUtils.repeat(null, 2) = null
+* StringUtils.repeat("", 0) = ""
+* StringUtils.repeat("", 2) = ""
+* StringUtils.repeat("a", 3) = "aaa"
+* StringUtils.repeat("ab", 2) = "abab"
+* StringUtils.repeat("a", -2) = ""
+*
+*
+* @param strthe String to repeat, may be null
+* @param repeat number of times to repeat str, negative treated as zero
+* @return a new String consisting of the original String repeated,
{@code null} if null String input
+*/
+ public static String repeat(final String str, final int repeat) {
+ // Performance tuned for 2.0 (JDK1.4)
+
+ if (str == null) {
+ return null;
+ }
+ if (repeat <= 0) {
+ return EMPTY;
+ }
+ final int inputLength = str.length();
+ if (repeat == 1 || inputLength == 0) {
+ return str;
+ }
+ if (inputLength == 1 && repeat <= PAD_LIMIT) {
+ return repeat(str.charAt(0), repeat);
+ }
+
+ final int outputLength = inputLength * repeat;
+ switch (inputLength) {
+ case 1:
+ return repeat(str.charAt(0), repeat);
+ case 2:
+ final char ch0 = str.charAt(0);
+ final char ch1 = str.charAt(1);
+ final char[] output2 = new char[outputLength];
+ for (int i = repeat * 2 - 2; i >= 0; i--, i--) {
+ output2[i] = ch0;
+ output2[i + 1] = ch1;
+ }
+ return new String(output2);
+ default:
+ final StringBuilder buf = new
StringBuilder(outpu
[jira] [Commented] (FLINK-10166) Dependency problems when executing SQL query in sql-client
[
https://issues.apache.org/jira/browse/FLINK-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669766#comment-16669766
]
ASF GitHub Bot commented on FLINK-10166:
igalshilman commented on issue #6966: [FLINK-10166] [table] Reduce dependencies
by removing org.apache.commons
URL: https://github.com/apache/flink/pull/6966#issuecomment-434604624
LGTM 👍
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Dependency problems when executing SQL query in sql-client
> --
>
> Key: FLINK-10166
> URL: https://issues.apache.org/jira/browse/FLINK-10166
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.6.0
>Reporter: Dawid Wysakowicz
>Assignee: Timo Walther
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> When tried to run query:
> {code}
> select count(distinct name) from (Values ('a'), ('b')) AS NameTable(name)
> {code}
> in {{sql-client.sh}} I got:
> {code}
> [ERROR] Could not execute SQL statement. Reason:
> org.codehaus.commons.compiler.CompileException: Line 43, Column 10: Unknown
> variable or type "org.apache.commons.codec.binary.Base64"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] twalthr commented on issue #6966: [FLINK-10166] [table] Reduce dependencies by removing org.apache.commons
twalthr commented on issue #6966: [FLINK-10166] [table] Reduce dependencies by removing org.apache.commons URL: https://github.com/apache/flink/pull/6966#issuecomment-434605691 Thank you @igalshilman. I will merge this... This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10166) Dependency problems when executing SQL query in sql-client
[
https://issues.apache.org/jira/browse/FLINK-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669767#comment-16669767
]
ASF GitHub Bot commented on FLINK-10166:
twalthr commented on issue #6966: [FLINK-10166] [table] Reduce dependencies by
removing org.apache.commons
URL: https://github.com/apache/flink/pull/6966#issuecomment-434605691
Thank you @igalshilman. I will merge this...
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Dependency problems when executing SQL query in sql-client
> --
>
> Key: FLINK-10166
> URL: https://issues.apache.org/jira/browse/FLINK-10166
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.6.0
>Reporter: Dawid Wysakowicz
>Assignee: Timo Walther
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> When tried to run query:
> {code}
> select count(distinct name) from (Values ('a'), ('b')) AS NameTable(name)
> {code}
> in {{sql-client.sh}} I got:
> {code}
> [ERROR] Could not execute SQL statement. Reason:
> org.codehaus.commons.compiler.CompileException: Line 43, Column 10: Unknown
> variable or type "org.apache.commons.codec.binary.Base64"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10732) force-shading 1.5.5 maven artifact was not released
Chesnay Schepler created FLINK-10732:
Summary: force-shading 1.5.5 maven artifact was not released
Key: FLINK-10732
URL: https://issues.apache.org/jira/browse/FLINK-10732
Project: Flink
Issue Type: Bug
Affects Versions: 1.5.5
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Fix For: 1.5.5
The 1.5.5 maven artifact for {{force-shading}} was not deployed. We have to
investigate whether other artifacts are missing as well and how to remedy the
problem, i.e. re-deploy the missing artifacts (which may or may not require a
separate vote).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10731) Support AVG on Date fields
[ https://issues.apache.org/jira/browse/FLINK-10731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669769#comment-16669769 ] xueyu commented on FLINK-10731: --- Could you please give your concrete use case, e.g. your sql or table api usage... > Support AVG on Date fields > -- > > Key: FLINK-10731 > URL: https://issues.apache.org/jira/browse/FLINK-10731 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Affects Versions: 1.6.2 >Reporter: Flavio Pompermaier >Priority: Minor > > AVG function does not work on date fields right now -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10728) Unexpected items on ASF mirrors
[
https://issues.apache.org/jira/browse/FLINK-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669770#comment-16669770
]
Chesnay Schepler commented on FLINK-10728:
--
{{flink-shaded}} is a collection of modified dependencies of Flink that don't
interfere with dependencies provided by a user. This was once part of Flink
itself, but was moved into a separate project to reduce complexity of the core
project and simplify it's dependency management.
Every Flink release (since 1.3 I believe) also contains {{flink-shaded}}.
According to the [ASF Release
Policy|https://www.apache.org/legal/release-policy.html#release-distribution]:
"_all artifacts MUST be uploaded to the project's subdirectory within the
canonical Apache distribution channel_". Not distributing said artifacts via
{{dist.apache.org}} should imply that {{flink-shaded}} is not released as an
official ASF release, but if it isn't an official ASF release we can't
distribute it at all via channels that would allow Flink to use it (i.e. maven
central).
> Unexpected items on ASF mirrors
> ---
>
> Key: FLINK-10728
> URL: https://issues.apache.org/jira/browse/FLINK-10728
> Project: Flink
> Issue Type: Bug
> Environment: https://www.apache.org/dist/flink/flink-shaded-4.0/
> https://www.apache.org/dist/flink/flink-shaded-5.0/
>Reporter: Sebb
>Assignee: Chesnay Schepler
>Priority: Major
>
> The flink-shaded artifacts are not linked from the download page, and don't
> appear to have been announced on any ASF mailing lists, so it's not clear why
> they are on the ASF mirrors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669772#comment-16669772
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on issue #6924: [FLINK-10600] Provide End-to-end test cases
for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#issuecomment-434608568
Please try adding `|| echo ""` after `awk '{print $1}'`, like in this
example:
```
set -x
set -e
set -u
set -o pipefail
PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}'|| echo "")
if [ ! -z "$PIDS" ]; then
echo "killing" $PIDS
else
echo "not killing" $PIDS
fi
echo "DONE"
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] pnowojski commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
pnowojski commented on issue #6924: [FLINK-10600] Provide End-to-end test cases
for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#issuecomment-434608568
Please try adding `|| echo ""` after `awk '{print $1}'`, like in this
example:
```
set -x
set -e
set -u
set -o pipefail
PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}'|| echo "")
if [ ! -z "$PIDS" ]; then
echo "killing" $PIDS
else
echo "not killing" $PIDS
fi
echo "DONE"
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] pnowojski edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
pnowojski edited a comment on issue #6924: [FLINK-10600] Provide End-to-end
test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#issuecomment-434608568
Please try adding `|| echo ""` after `awk '{print $1}'`, like in this
example:
```
set -x
set -e
set -u
set -o pipefail
PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}'|| echo "")
if [ ! -z "$PIDS" ]; then
echo "killing" $PIDS
else
echo "not killing" $PIDS
fi
echo "DONE"
```
if it works, then add comment above
```
# Terminate QuorumPeerMain process if it still exists
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669774#comment-16669774
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski edited a comment on issue #6924: [FLINK-10600] Provide End-to-end
test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#issuecomment-434608568
Please try adding `|| echo ""` after `awk '{print $1}'`, like in this
example:
```
set -x
set -e
set -u
set -o pipefail
PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}'|| echo "")
if [ ! -z "$PIDS" ]; then
echo "killing" $PIDS
else
echo "not killing" $PIDS
fi
echo "DONE"
```
if it works, then add comment above
```
# Terminate QuorumPeerMain process if it still exists
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-9050) Expose operator IO counter metrics
[ https://issues.apache.org/jira/browse/FLINK-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining reassigned FLINK-9050: - Assignee: lining (was: Chesnay Schepler) > Expose operator IO counter metrics > -- > > Key: FLINK-9050 > URL: https://issues.apache.org/jira/browse/FLINK-9050 > Project: Flink > Issue Type: New Feature > Components: DataSet API, DataStream API, Metrics >Reporter: Chesnay Schepler >Assignee: lining >Priority: Major > Fix For: 1.7.0 > > > To properly expose the number of records read by sources / emitted by sink we > have to expose the operator counter metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-7286) Flink Dashboard fails to display bytes/records received by sources
[ https://issues.apache.org/jira/browse/FLINK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669776#comment-16669776 ] lining commented on FLINK-7286: --- Ok， thanks. > Flink Dashboard fails to display bytes/records received by sources > -- > > Key: FLINK-7286 > URL: https://issues.apache.org/jira/browse/FLINK-7286 > Project: Flink > Issue Type: Improvement > Components: Metrics, Webfrontend >Affects Versions: 1.3.1 >Reporter: Elias Levy >Assignee: Hai Zhou >Priority: Major > Fix For: 1.7.0 > > > It appears Flink can't measure the number of bytes read or records produced > by a source (e.g. Kafka source). This is particularly problematic for simple > jobs where the job pipeline is chained, and in which there are no > measurements between operators. Thus, in the UI it appears that the job is > not consuming any data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10732) force-shading 1.5.5 maven artifact was not released
[
https://issues.apache.org/jira/browse/FLINK-10732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669779#comment-16669779
]
Chesnay Schepler commented on FLINK-10732:
--
I've checked all artifacts at {{repository.apache.org}} and {{force-shading}}
is indeed the only missing artifact.
> force-shading 1.5.5 maven artifact was not released
> ---
>
> Key: FLINK-10732
> URL: https://issues.apache.org/jira/browse/FLINK-10732
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.5.5
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Fix For: 1.5.5
>
>
> The 1.5.5 maven artifact for {{force-shading}} was not deployed. We have to
> investigate whether other artifacts are missing as well and how to remedy the
> problem, i.e. re-deploy the missing artifacts (which may or may not require a
> separate vote).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10730) How to resume Flink SQL Job ?
[
https://issues.apache.org/jira/browse/FLINK-10730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669783#comment-16669783
]
Timo Walther commented on FLINK-10730:
--
[~zhangxin0112zx] please use the mailing lists for asking questions. JIRA
issues are meant for tracking bugs or new features. I will close this issue.
Regarding your question, you can use all regular Flink [command line
arguments|https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/cli.html#usage]
in the {{deployment}} section of an environment file. In theory,
{{fromSavepoint: "/path/to/something"}} should do the trick, but this is not
officially supported and tested. Btw also your {{-u}} parameter is not official
API.
> How to resume Flink SQL Job ?
> -
>
> Key: FLINK-10730
> URL: https://issues.apache.org/jira/browse/FLINK-10730
> Project: Flink
> Issue Type: Improvement
> Components: SQL Client
>Affects Versions: 1.6.2
>Reporter: xinzhang
>Priority: Major
>
> Hi. Is any way to resume Flink SQL Job ? The guide here [flink
> title|https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/savepoints.html#resuming-from-savepoints]
> means flink resume Flink Job (Like some Jar) but no Flink SQL Job.
> 1.Now I submit the Flink SQL Job use the cmd：
> cat test.sql |xargs -0 $FLINK_HOME/bin/sql-client.sh embedded -e
> sql-client-kafka-json.yaml --library $FLINK_HOME/lib -u
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-10730) How to resume Flink SQL Job ?
[ https://issues.apache.org/jira/browse/FLINK-10730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther closed FLINK-10730. Resolution: Invalid > How to resume Flink SQL Job ? > - > > Key: FLINK-10730 > URL: https://issues.apache.org/jira/browse/FLINK-10730 > Project: Flink > Issue Type: Improvement > Components: SQL Client >Affects Versions: 1.6.2 >Reporter: xinzhang >Priority: Major > > Hi. Is any way to resume Flink SQL Job ? The guide here [flink > title|https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/savepoints.html#resuming-from-savepoints] > means flink resume Flink Job (Like some Jar) but no Flink SQL Job. > 1.Now I submit the Flink SQL Job use the cmd： > cat test.sql |xargs -0 $FLINK_HOME/bin/sql-client.sh embedded -e > sql-client-kafka-json.yaml --library $FLINK_HOME/lib -u > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10166) Dependency problems when executing SQL query in sql-client
[
https://issues.apache.org/jira/browse/FLINK-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669786#comment-16669786
]
Timo Walther commented on FLINK-10166:
--
Fixed in 1.7.0: 08d875eadec779f68baa1cc7003dab7bdd51640a
> Dependency problems when executing SQL query in sql-client
> --
>
> Key: FLINK-10166
> URL: https://issues.apache.org/jira/browse/FLINK-10166
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.6.0
>Reporter: Dawid Wysakowicz
>Assignee: Timo Walther
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> When tried to run query:
> {code}
> select count(distinct name) from (Values ('a'), ('b')) AS NameTable(name)
> {code}
> in {{sql-client.sh}} I got:
> {code}
> [ERROR] Could not execute SQL statement. Reason:
> org.codehaus.commons.compiler.CompileException: Line 43, Column 10: Unknown
> variable or type "org.apache.commons.codec.binary.Base64"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] asfgit closed pull request #6966: [FLINK-10166] [table] Reduce dependencies by removing org.apache.commons
asfgit closed pull request #6966: [FLINK-10166] [table] Reduce dependencies by
removing org.apache.commons
URL: https://github.com/apache/flink/pull/6966
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/DescriptorProperties.java
b/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/DescriptorProperties.java
index 0d2f835958f..c4769814728 100644
---
a/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/DescriptorProperties.java
+++
b/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/DescriptorProperties.java
@@ -1326,7 +1326,7 @@ private int extractMaxIndex(String key, String
suffixPattern) {
}
public static String toString(String str) {
- return EncodingUtils.escapeJava(str).replace("\\/", "/"); //
'/' must not be escaped
+ return EncodingUtils.escapeJava(str);
}
public static String toString(String key, String value) {
diff --git
a/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
b/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
index 41fa58ef055..47aac25e897 100644
---
a/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
+++
b/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
@@ -19,6 +19,7 @@
package org.apache.flink.table.utils;
import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableException;
import org.apache.flink.table.api.ValidationException;
import org.apache.flink.util.InstantiationUtil;
@@ -26,9 +27,12 @@
import java.io.Serializable;
import java.io.StringWriter;
import java.io.Writer;
-import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
import java.util.Base64;
+import static java.nio.charset.StandardCharsets.UTF_8;
+
/**
* General utilities for string-encoding. This class is used to avoid
additional dependencies
* to other projects.
@@ -40,6 +44,8 @@
private static final Base64.Decoder BASE64_DECODER =
java.util.Base64.getUrlDecoder();
+ private static final char[] HEX_CHARS =
"0123456789abcdef".toCharArray();
+
private EncodingUtils() {
// do not instantiate
}
@@ -47,7 +53,7 @@ private EncodingUtils() {
public static String encodeObjectToString(Serializable obj) {
try {
final byte[] bytes =
InstantiationUtil.serializeObject(obj);
- return new String(BASE64_ENCODER.encode(bytes),
StandardCharsets.UTF_8);
+ return new String(BASE64_ENCODER.encode(bytes), UTF_8);
} catch (Exception e) {
throw new ValidationException(
"Unable to serialize object '" + obj.toString()
+ "' of class '" + obj.getClass().getName() + "'.");
@@ -60,7 +66,7 @@ public static String encodeObjectToString(Serializable obj) {
public static T decodeStringToObject(String
base64String, Class baseClass, ClassLoader classLoader) {
try {
- final byte[] bytes =
BASE64_DECODER.decode(base64String.getBytes(StandardCharsets.UTF_8));
+ final byte[] bytes =
BASE64_DECODER.decode(base64String.getBytes(UTF_8));
final T instance =
InstantiationUtil.deserializeObject(bytes, classLoader);
if (instance != null &&
!baseClass.isAssignableFrom(instance.getClass())) {
throw new ValidationException(
@@ -87,10 +93,138 @@ public static String encodeObjectToString(Serializable
obj) {
return loadClass(qualifiedName,
Thread.currentThread().getContextClassLoader());
}
+ public static String encodeStringToBase64(String string) {
+ return new
String(java.util.Base64.getEncoder().encode(string.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static String decodeBase64ToString(String base64) {
+ return new
String(java.util.Base64.getDecoder().decode(base64.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static byte[] md5(String string) {
+ try {
+ return
MessageDigest.getInstance("MD5").digest(string.getBytes(UTF_8));
+ } catch (NoSuchAlgorithmException e) {
+ throw new TableException("Unsupported MD5 algorithm.",
e);
+ }
+ }
+
[jira] [Commented] (FLINK-10704) Fix sql client end to end test failure
[
https://issues.apache.org/jira/browse/FLINK-10704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669791#comment-16669791
]
Piotr Nowojski commented on FLINK-10704:
[~yanghua] I don't understand this:
> I do not agree to filter out `WARN` lines because of warning logs cannot be
> used as a measure of failure
what do you mean? If as you say, "warning logs cannot be used as a measure of
failure", then why not filter them out as I proposed?
Btw, this global `check_logs_for_errors ` is broken design. We might want to
have some limited global list of exclusions, but adding more entries to ever
growing list is unmaintainable. I think we should filter out this one specific
warning in kafka tests and do not mess with changing global rules for all of
the tests.
> Fix sql client end to end test failure
> --
>
> Key: FLINK-10704
> URL: https://issues.apache.org/jira/browse/FLINK-10704
> Project: Flink
> Issue Type: Bug
> Components: E2E Tests, Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
>
> The log file contains the following sentence:
> {code:java}
> 2018-10-29 03:27:39,209 WARN
> org.apache.flink.kafka010.shaded.org.apache.kafka.common.utils.AppInfoParser
> - Error while loading kafka-version.properties :null
> {code}
> The reason for this log is that we explicitly exclude the version description
> file of the kafka client when packaging the connector:
> {code:java}
>
>
> *:*
>
> kafka/kafka-version.properties
>
>
> {code}
> When the shell scan the "error" keyword with grep, it will hit, so the test
> will fail.
> {code:java}
> function check_logs_for_errors {
> error_count=$(grep -rv "GroupCoordinatorNotAvailableException"
> $FLINK_DIR/log \
> | grep -v "RetriableCommitFailedException" \
> | grep -v "NoAvailableBrokersException" \
> | grep -v "Async Kafka commit failed" \
> | grep -v "DisconnectException" \
> | grep -v "AskTimeoutException" \
> | grep -v "WARN akka.remote.transport.netty.NettyTransport" \
> | grep -v "WARN
> org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline" \
> | grep -v "jvm-exit-on-fatal-error" \
> | grep -v '^INFO:.*AWSErrorCode=\[400 Bad
> Request\].*ServiceEndpoint=\[https://.*\.s3\.amazonaws\.com\].*RequestType=\[HeadBucketRequest\]'
> \
> | grep -v "RejectedExecutionException" \
> | grep -v "An exception was thrown by an exception handler" \
> | grep -v "java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/exceptions/YarnException" \
> | grep -v "java.lang.NoClassDefFoundError:
> org/apache/hadoop/conf/Configuration" \
> | grep -v
> "org.apache.flink.fs.shaded.hadoop3.org.apache.commons.beanutils.FluentPropertyBeanIntrospector
> - Error when creating PropertyDescriptor for public final void
> org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
> Ignoring this property." \
> | grep -ic "error")//here
> if [[ ${error_count} -gt 0 ]]; then
> echo "Found error in log files:"
> cat $FLINK_DIR/log/*
> EXIT_CODE=1
> fi
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10166) Dependency problems when executing SQL query in sql-client
[
https://issues.apache.org/jira/browse/FLINK-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669790#comment-16669790
]
ASF GitHub Bot commented on FLINK-10166:
asfgit closed pull request #6966: [FLINK-10166] [table] Reduce dependencies by
removing org.apache.commons
URL: https://github.com/apache/flink/pull/6966
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/DescriptorProperties.java
b/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/DescriptorProperties.java
index 0d2f835958f..c4769814728 100644
---
a/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/DescriptorProperties.java
+++
b/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/DescriptorProperties.java
@@ -1326,7 +1326,7 @@ private int extractMaxIndex(String key, String
suffixPattern) {
}
public static String toString(String str) {
- return EncodingUtils.escapeJava(str).replace("\\/", "/"); //
'/' must not be escaped
+ return EncodingUtils.escapeJava(str);
}
public static String toString(String key, String value) {
diff --git
a/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
b/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
index 41fa58ef055..47aac25e897 100644
---
a/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
+++
b/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/utils/EncodingUtils.java
@@ -19,6 +19,7 @@
package org.apache.flink.table.utils;
import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.TableException;
import org.apache.flink.table.api.ValidationException;
import org.apache.flink.util.InstantiationUtil;
@@ -26,9 +27,12 @@
import java.io.Serializable;
import java.io.StringWriter;
import java.io.Writer;
-import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
import java.util.Base64;
+import static java.nio.charset.StandardCharsets.UTF_8;
+
/**
* General utilities for string-encoding. This class is used to avoid
additional dependencies
* to other projects.
@@ -40,6 +44,8 @@
private static final Base64.Decoder BASE64_DECODER =
java.util.Base64.getUrlDecoder();
+ private static final char[] HEX_CHARS =
"0123456789abcdef".toCharArray();
+
private EncodingUtils() {
// do not instantiate
}
@@ -47,7 +53,7 @@ private EncodingUtils() {
public static String encodeObjectToString(Serializable obj) {
try {
final byte[] bytes =
InstantiationUtil.serializeObject(obj);
- return new String(BASE64_ENCODER.encode(bytes),
StandardCharsets.UTF_8);
+ return new String(BASE64_ENCODER.encode(bytes), UTF_8);
} catch (Exception e) {
throw new ValidationException(
"Unable to serialize object '" + obj.toString()
+ "' of class '" + obj.getClass().getName() + "'.");
@@ -60,7 +66,7 @@ public static String encodeObjectToString(Serializable obj) {
public static T decodeStringToObject(String
base64String, Class baseClass, ClassLoader classLoader) {
try {
- final byte[] bytes =
BASE64_DECODER.decode(base64String.getBytes(StandardCharsets.UTF_8));
+ final byte[] bytes =
BASE64_DECODER.decode(base64String.getBytes(UTF_8));
final T instance =
InstantiationUtil.deserializeObject(bytes, classLoader);
if (instance != null &&
!baseClass.isAssignableFrom(instance.getClass())) {
throw new ValidationException(
@@ -87,10 +93,138 @@ public static String encodeObjectToString(Serializable
obj) {
return loadClass(qualifiedName,
Thread.currentThread().getContextClassLoader());
}
+ public static String encodeStringToBase64(String string) {
+ return new
String(java.util.Base64.getEncoder().encode(string.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static String decodeBase64ToString(String base64) {
+ return new
String(java.util.Base64.getDecoder().decode(base64.getBytes(UTF_8)), UTF_8);
+ }
+
+ public static byte[] md5(String string) {
+ try {
+
[GitHub] pnowojski commented on a change in pull request #6963: [FLINK-10704] Fix sql client end to end test failure
pnowojski commented on a change in pull request #6963: [FLINK-10704] Fix sql
client end to end test failure
URL: https://github.com/apache/flink/pull/6963#discussion_r229610938
##
File path: flink-end-to-end-tests/test-scripts/common.sh
##
@@ -316,7 +316,7 @@ function check_logs_for_errors {
| grep -v "java.lang.NoClassDefFoundError:
org/apache/hadoop/yarn/exceptions/YarnException" \
| grep -v "java.lang.NoClassDefFoundError:
org/apache/hadoop/conf/Configuration" \
| grep -v
"org.apache.flink.fs.shaded.hadoop3.org.apache.commons.beanutils.FluentPropertyBeanIntrospector
- Error when creating PropertyDescriptor for public final void
org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
Ignoring this property." \
- | grep -ic "error")
+ | grep -c "ERROR")
Review comment:
As I have written in the ticket:
> This global `check_logs_for_errors ` is broken design. We might want to
have some limited global list of exclusions, but adding more entries to ever
growing list is unmaintainable. I think we should filter out this one specific
warning in kafka tests and do not mess with changing global rules for all of
the tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-10704) Fix sql client end to end test failure
[
https://issues.apache.org/jira/browse/FLINK-10704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669794#comment-16669794
]
ASF GitHub Bot commented on FLINK-10704:
pnowojski commented on a change in pull request #6963: [FLINK-10704] Fix sql
client end to end test failure
URL: https://github.com/apache/flink/pull/6963#discussion_r229610938
##
File path: flink-end-to-end-tests/test-scripts/common.sh
##
@@ -316,7 +316,7 @@ function check_logs_for_errors {
| grep -v "java.lang.NoClassDefFoundError:
org/apache/hadoop/yarn/exceptions/YarnException" \
| grep -v "java.lang.NoClassDefFoundError:
org/apache/hadoop/conf/Configuration" \
| grep -v
"org.apache.flink.fs.shaded.hadoop3.org.apache.commons.beanutils.FluentPropertyBeanIntrospector
- Error when creating PropertyDescriptor for public final void
org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
Ignoring this property." \
- | grep -ic "error")
+ | grep -c "ERROR")
Review comment:
As I have written in the ticket:
> This global `check_logs_for_errors ` is broken design. We might want to
have some limited global list of exclusions, but adding more entries to ever
growing list is unmaintainable. I think we should filter out this one specific
warning in kafka tests and do not mess with changing global rules for all of
the tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fix sql client end to end test failure
> --
>
> Key: FLINK-10704
> URL: https://issues.apache.org/jira/browse/FLINK-10704
> Project: Flink
> Issue Type: Bug
> Components: E2E Tests, Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
>
> The log file contains the following sentence:
> {code:java}
> 2018-10-29 03:27:39,209 WARN
> org.apache.flink.kafka010.shaded.org.apache.kafka.common.utils.AppInfoParser
> - Error while loading kafka-version.properties :null
> {code}
> The reason for this log is that we explicitly exclude the version description
> file of the kafka client when packaging the connector:
> {code:java}
>
>
> *:*
>
> kafka/kafka-version.properties
>
>
> {code}
> When the shell scan the "error" keyword with grep, it will hit, so the test
> will fail.
> {code:java}
> function check_logs_for_errors {
> error_count=$(grep -rv "GroupCoordinatorNotAvailableException"
> $FLINK_DIR/log \
> | grep -v "RetriableCommitFailedException" \
> | grep -v "NoAvailableBrokersException" \
> | grep -v "Async Kafka commit failed" \
> | grep -v "DisconnectException" \
> | grep -v "AskTimeoutException" \
> | grep -v "WARN akka.remote.transport.netty.NettyTransport" \
> | grep -v "WARN
> org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline" \
> | grep -v "jvm-exit-on-fatal-error" \
> | grep -v '^INFO:.*AWSErrorCode=\[400 Bad
> Request\].*ServiceEndpoint=\[https://.*\.s3\.amazonaws\.com\].*RequestType=\[HeadBucketRequest\]'
> \
> | grep -v "RejectedExecutionException" \
> | grep -v "An exception was thrown by an exception handler" \
> | grep -v "java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/exceptions/YarnException" \
> | grep -v "java.lang.NoClassDefFoundError:
> org/apache/hadoop/conf/Configuration" \
> | grep -v
> "org.apache.flink.fs.shaded.hadoop3.org.apache.commons.beanutils.FluentPropertyBeanIntrospector
> - Error when creating PropertyDescriptor for public final void
> org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
> Ignoring this property." \
> | grep -ic "error")//here
> if [[ ${error_count} -gt 0 ]]; then
> echo "Found error in log files:"
> cat $FLINK_DIR/log/*
> EXIT_CODE=1
> fi
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly.
zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r229611970
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/util/FileOffsetRange.java
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.util;
+
+import java.io.File;
+import java.io.Serializable;
+
+/**
+ * FileOffsetRange is used to decide which part of files to read.
+ */
+public class FileOffsetRange implements Serializable{
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[
https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669799#comment-16669799
]
ASF GitHub Bot commented on FLINK-10002:
zhangxinyu1 commented on a change in pull request #6770: [FLINK-10002]
[Webfrontend] WebUI shows jm/tm logs more friendly.
URL: https://github.com/apache/flink/pull/6770#discussion_r229611970
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/util/FileOffsetRange.java
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.util;
+
+import java.io.File;
+import java.io.Serializable;
+
+/**
+ * FileOffsetRange is used to decide which part of files to read.
+ */
+public class FileOffsetRange implements Serializable{
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> WebUI shows logs unfriendly, especially when the amount of logs is large
>
>
> Key: FLINK-10002
> URL: https://issues.apache.org/jira/browse/FLINK-10002
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: zhangxinyu
>Assignee: zhangxinyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Attachments: image-2018-09-10-11-38-07-973.png
>
>
> When a streaming job run for a long time, the amount of logs may be very
> large. The current WebUI shows all content of logs. It will cost much time to
> download logs from task managers. and the browser cannot display the logs.
> Therefore, I suggest that Flink uses DailyRollingAppender to split logs by
> default, and task manager provides an API that can get logs based on a
> parameter of time interval. In this way WebUI can display logs based on time
> interval.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] twalthr closed pull request #6863: [FLINK-10166][table] Replace commons.codec.binary.Base64 with java.util.Base64
twalthr closed pull request #6863: [FLINK-10166][table] Replace
commons.codec.binary.Base64 with java.util.Base64
URL: https://github.com/apache/flink/pull/6863
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/codegen/AggregationCodeGenerator.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/codegen/AggregationCodeGenerator.scala
index 11c00080329..c7a6a64ab94 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/codegen/AggregationCodeGenerator.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/codegen/AggregationCodeGenerator.scala
@@ -21,7 +21,6 @@ import java.lang.reflect.Modifier
import java.lang.{Iterable => JIterable}
import org.apache.calcite.rex.RexLiteral
-import org.apache.commons.codec.binary.Base64
import org.apache.flink.api.common.state.{ListStateDescriptor,
MapStateDescriptor, State, StateDescriptor}
import org.apache.flink.api.common.typeinfo.{BasicTypeInfo, TypeInformation}
import org.apache.flink.api.java.typeutils.RowTypeInfo
@@ -330,7 +329,7 @@ class AggregationCodeGenerator(
s"""
|$descClassQualifier $descFieldTerm = ($descClassQualifier)
| org.apache.flink.util.InstantiationUtil.deserializeObject(
- |
org.apache.commons.codec.binary.Base64.decodeBase64("$serializedData"),
+ |
java.util.Base64.getUrlDecoder().decode("$serializedData".getBytes("UTF-8")),
| $contextTerm.getUserCodeClassLoader());
|""".stripMargin
val createDataView = if (dataViewField.getType == classOf[MapView[_,
_]]) {
@@ -774,6 +773,6 @@ class AggregationCodeGenerator(
@throws[Exception]
def serializeStateDescriptor(stateDescriptor: StateDescriptor[_, _]): String
= {
val byteArray = InstantiationUtil.serializeObject(stateDescriptor)
-Base64.encodeBase64URLSafeString(byteArray)
+new
String(java.util.Base64.getUrlEncoder.withoutPadding().encode(byteArray),
"UTF-8")
}
}
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/descriptors/DescriptorProperties.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/descriptors/DescriptorProperties.scala
index 9328c806913..4177c3a1620 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/descriptors/DescriptorProperties.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/descriptors/DescriptorProperties.scala
@@ -26,7 +26,6 @@ import java.util.function.{Consumer, Supplier}
import java.util.regex.Pattern
import java.util.{Optional, List => JList, Map => JMap}
-import org.apache.commons.codec.binary.Base64
import org.apache.commons.lang.StringEscapeUtils
import org.apache.flink.api.common.typeinfo.TypeInformation
import org.apache.flink.api.java.tuple.{Tuple2 => JTuple2}
@@ -1356,7 +1355,7 @@ object DescriptorProperties {
}
try {
val byteArray = InstantiationUtil.serializeObject(obj)
- Base64.encodeBase64URLSafeString(byteArray)
+ new
String(java.util.Base64.getUrlEncoder.withoutPadding().encode(byteArray),
"UTF-8")
} catch {
case e: Exception =>
throw new ValidationException(
@@ -1366,7 +1365,7 @@ object DescriptorProperties {
def deserialize[T](data: String, expected: Class[T]): T = {
try {
- val byteData = Base64.decodeBase64(data)
+ val byteData =
java.util.Base64.getUrlDecoder.decode(data.getBytes("UTF-8"))
val obj = InstantiationUtil.deserializeObject[T](
byteData,
Thread.currentThread.getContextClassLoader)
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala
index 2c08001d20c..7e5128cf939 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala
@@ -24,7 +24,6 @@ import java.lang.{Integer => JInt, Long => JLong}
import java.lang.reflect.{Method, Modifier}
import java.sql.{Date, Time, Timestamp}
-import org.apache.commons.codec.binary.Base64
import com.google.common.primitives.Primitives
import org.apache.calcite.rel.`type`.RelDataType
import org.apache.calcite.sql.`type`.SqlOperandTypeChecker.Consistency
@@ -736,12 +735,12 @@ object UserDefinedFunctionUtils {
@throws[Exception]
def serialize(function: UserDefinedFunction): String = {
val byteArray = Instanti
[jira] [Commented] (FLINK-10166) Dependency problems when executing SQL query in sql-client
[
https://issues.apache.org/jira/browse/FLINK-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669805#comment-16669805
]
ASF GitHub Bot commented on FLINK-10166:
twalthr commented on issue #6863: [FLINK-10166][table] Replace
commons.codec.binary.Base64 with java.util.Base64
URL: https://github.com/apache/flink/pull/6863#issuecomment-434614175
I tried to fix this issue once and for all in #6966. This PR had to be
rebased to the new `flink-table-common` module anyway. I will close this PR.
Sorry, for the inconvenience.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Dependency problems when executing SQL query in sql-client
> --
>
> Key: FLINK-10166
> URL: https://issues.apache.org/jira/browse/FLINK-10166
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.6.0
>Reporter: Dawid Wysakowicz
>Assignee: Timo Walther
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> When tried to run query:
> {code}
> select count(distinct name) from (Values ('a'), ('b')) AS NameTable(name)
> {code}
> in {{sql-client.sh}} I got:
> {code}
> [ERROR] Could not execute SQL statement. Reason:
> org.codehaus.commons.compiler.CompileException: Line 43, Column 10: Unknown
> variable or type "org.apache.commons.codec.binary.Base64"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] twalthr commented on issue #6863: [FLINK-10166][table] Replace commons.codec.binary.Base64 with java.util.Base64
twalthr commented on issue #6863: [FLINK-10166][table] Replace commons.codec.binary.Base64 with java.util.Base64 URL: https://github.com/apache/flink/pull/6863#issuecomment-434614175 I tried to fix this issue once and for all in #6966. This PR had to be rebased to the new `flink-table-common` module anyway. I will close this PR. Sorry, for the inconvenience. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10166) Dependency problems when executing SQL query in sql-client
[
https://issues.apache.org/jira/browse/FLINK-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669806#comment-16669806
]
ASF GitHub Bot commented on FLINK-10166:
twalthr closed pull request #6863: [FLINK-10166][table] Replace
commons.codec.binary.Base64 with java.util.Base64
URL: https://github.com/apache/flink/pull/6863
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/codegen/AggregationCodeGenerator.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/codegen/AggregationCodeGenerator.scala
index 11c00080329..c7a6a64ab94 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/codegen/AggregationCodeGenerator.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/codegen/AggregationCodeGenerator.scala
@@ -21,7 +21,6 @@ import java.lang.reflect.Modifier
import java.lang.{Iterable => JIterable}
import org.apache.calcite.rex.RexLiteral
-import org.apache.commons.codec.binary.Base64
import org.apache.flink.api.common.state.{ListStateDescriptor,
MapStateDescriptor, State, StateDescriptor}
import org.apache.flink.api.common.typeinfo.{BasicTypeInfo, TypeInformation}
import org.apache.flink.api.java.typeutils.RowTypeInfo
@@ -330,7 +329,7 @@ class AggregationCodeGenerator(
s"""
|$descClassQualifier $descFieldTerm = ($descClassQualifier)
| org.apache.flink.util.InstantiationUtil.deserializeObject(
- |
org.apache.commons.codec.binary.Base64.decodeBase64("$serializedData"),
+ |
java.util.Base64.getUrlDecoder().decode("$serializedData".getBytes("UTF-8")),
| $contextTerm.getUserCodeClassLoader());
|""".stripMargin
val createDataView = if (dataViewField.getType == classOf[MapView[_,
_]]) {
@@ -774,6 +773,6 @@ class AggregationCodeGenerator(
@throws[Exception]
def serializeStateDescriptor(stateDescriptor: StateDescriptor[_, _]): String
= {
val byteArray = InstantiationUtil.serializeObject(stateDescriptor)
-Base64.encodeBase64URLSafeString(byteArray)
+new
String(java.util.Base64.getUrlEncoder.withoutPadding().encode(byteArray),
"UTF-8")
}
}
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/descriptors/DescriptorProperties.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/descriptors/DescriptorProperties.scala
index 9328c806913..4177c3a1620 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/descriptors/DescriptorProperties.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/descriptors/DescriptorProperties.scala
@@ -26,7 +26,6 @@ import java.util.function.{Consumer, Supplier}
import java.util.regex.Pattern
import java.util.{Optional, List => JList, Map => JMap}
-import org.apache.commons.codec.binary.Base64
import org.apache.commons.lang.StringEscapeUtils
import org.apache.flink.api.common.typeinfo.TypeInformation
import org.apache.flink.api.java.tuple.{Tuple2 => JTuple2}
@@ -1356,7 +1355,7 @@ object DescriptorProperties {
}
try {
val byteArray = InstantiationUtil.serializeObject(obj)
- Base64.encodeBase64URLSafeString(byteArray)
+ new
String(java.util.Base64.getUrlEncoder.withoutPadding().encode(byteArray),
"UTF-8")
} catch {
case e: Exception =>
throw new ValidationException(
@@ -1366,7 +1365,7 @@ object DescriptorProperties {
def deserialize[T](data: String, expected: Class[T]): T = {
try {
- val byteData = Base64.decodeBase64(data)
+ val byteData =
java.util.Base64.getUrlDecoder.decode(data.getBytes("UTF-8"))
val obj = InstantiationUtil.deserializeObject[T](
byteData,
Thread.currentThread.getContextClassLoader)
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala
index 2c08001d20c..7e5128cf939 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/utils/UserDefinedFunctionUtils.scala
@@ -24,7 +24,6 @@ import java.lang.{Integer => JInt, Long => JLong}
import java.lang.reflect.{Method, Modifier}
import java.sql.{Date, Time, Timestamp}
-import org.apache.commons.codec.binary.Base64
import com.google.common.primitives.Primitives
import org.apache.calcite.re
[GitHub] yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434615379 @pnowojski Thank you for your guidance, now the test can pass. I submitted the last change, can you help me to review it again? Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669808#comment-16669808 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434615379 @pnowojski Thank you for your guidance, now the test can pass. I submitted the last change, can you help me to review it again? Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10704) Fix sql client end to end test failure
[
https://issues.apache.org/jira/browse/FLINK-10704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669810#comment-16669810
]
ASF GitHub Bot commented on FLINK-10704:
yanghua commented on a change in pull request #6963: [FLINK-10704] Fix sql
client end to end test failure
URL: https://github.com/apache/flink/pull/6963#discussion_r229615215
##
File path: flink-end-to-end-tests/test-scripts/common.sh
##
@@ -316,7 +316,7 @@ function check_logs_for_errors {
| grep -v "java.lang.NoClassDefFoundError:
org/apache/hadoop/yarn/exceptions/YarnException" \
| grep -v "java.lang.NoClassDefFoundError:
org/apache/hadoop/conf/Configuration" \
| grep -v
"org.apache.flink.fs.shaded.hadoop3.org.apache.commons.beanutils.FluentPropertyBeanIntrospector
- Error when creating PropertyDescriptor for public final void
org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
Ignoring this property." \
- | grep -ic "error")
+ | grep -c "ERROR")
Review comment:
Agree. I thought that you want to filter the warning log globally. Sorry, I
misunderstood what you mean.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fix sql client end to end test failure
> --
>
> Key: FLINK-10704
> URL: https://issues.apache.org/jira/browse/FLINK-10704
> Project: Flink
> Issue Type: Bug
> Components: E2E Tests, Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
>
> The log file contains the following sentence:
> {code:java}
> 2018-10-29 03:27:39,209 WARN
> org.apache.flink.kafka010.shaded.org.apache.kafka.common.utils.AppInfoParser
> - Error while loading kafka-version.properties :null
> {code}
> The reason for this log is that we explicitly exclude the version description
> file of the kafka client when packaging the connector:
> {code:java}
>
>
> *:*
>
> kafka/kafka-version.properties
>
>
> {code}
> When the shell scan the "error" keyword with grep, it will hit, so the test
> will fail.
> {code:java}
> function check_logs_for_errors {
> error_count=$(grep -rv "GroupCoordinatorNotAvailableException"
> $FLINK_DIR/log \
> | grep -v "RetriableCommitFailedException" \
> | grep -v "NoAvailableBrokersException" \
> | grep -v "Async Kafka commit failed" \
> | grep -v "DisconnectException" \
> | grep -v "AskTimeoutException" \
> | grep -v "WARN akka.remote.transport.netty.NettyTransport" \
> | grep -v "WARN
> org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline" \
> | grep -v "jvm-exit-on-fatal-error" \
> | grep -v '^INFO:.*AWSErrorCode=\[400 Bad
> Request\].*ServiceEndpoint=\[https://.*\.s3\.amazonaws\.com\].*RequestType=\[HeadBucketRequest\]'
> \
> | grep -v "RejectedExecutionException" \
> | grep -v "An exception was thrown by an exception handler" \
> | grep -v "java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/exceptions/YarnException" \
> | grep -v "java.lang.NoClassDefFoundError:
> org/apache/hadoop/conf/Configuration" \
> | grep -v
> "org.apache.flink.fs.shaded.hadoop3.org.apache.commons.beanutils.FluentPropertyBeanIntrospector
> - Error when creating PropertyDescriptor for public final void
> org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
> Ignoring this property." \
> | grep -ic "error")//here
> if [[ ${error_count} -gt 0 ]]; then
> echo "Found error in log files:"
> cat $FLINK_DIR/log/*
> EXIT_CODE=1
> fi
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] yanghua commented on a change in pull request #6963: [FLINK-10704] Fix sql client end to end test failure
yanghua commented on a change in pull request #6963: [FLINK-10704] Fix sql
client end to end test failure
URL: https://github.com/apache/flink/pull/6963#discussion_r229615215
##
File path: flink-end-to-end-tests/test-scripts/common.sh
##
@@ -316,7 +316,7 @@ function check_logs_for_errors {
| grep -v "java.lang.NoClassDefFoundError:
org/apache/hadoop/yarn/exceptions/YarnException" \
| grep -v "java.lang.NoClassDefFoundError:
org/apache/hadoop/conf/Configuration" \
| grep -v
"org.apache.flink.fs.shaded.hadoop3.org.apache.commons.beanutils.FluentPropertyBeanIntrospector
- Error when creating PropertyDescriptor for public final void
org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
Ignoring this property." \
- | grep -ic "error")
+ | grep -c "ERROR")
Review comment:
Agree. I thought that you want to filter the warning log globally. Sorry, I
misunderstood what you mean.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-10166) Dependency problems when executing SQL query in sql-client
[
https://issues.apache.org/jira/browse/FLINK-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther updated FLINK-10166:
-
Fix Version/s: 1.6.3
> Dependency problems when executing SQL query in sql-client
> --
>
> Key: FLINK-10166
> URL: https://issues.apache.org/jira/browse/FLINK-10166
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.6.0
>Reporter: Dawid Wysakowicz
>Assignee: Timo Walther
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.6.3, 1.7.0
>
>
> When tried to run query:
> {code}
> select count(distinct name) from (Values ('a'), ('b')) AS NameTable(name)
> {code}
> in {{sql-client.sh}} I got:
> {code}
> [ERROR] Could not execute SQL statement. Reason:
> org.codehaus.commons.compiler.CompileException: Line 43, Column 10: Unknown
> variable or type "org.apache.commons.codec.binary.Base64"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9050) Expose operator IO counter metrics
[ https://issues.apache.org/jira/browse/FLINK-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669829#comment-16669829 ] lining commented on FLINK-9050: --- Hi, [~wolli]. add new rest api for operator or add this metric to */jobs/:jobid* > Expose operator IO counter metrics > -- > > Key: FLINK-9050 > URL: https://issues.apache.org/jira/browse/FLINK-9050 > Project: Flink > Issue Type: New Feature > Components: DataSet API, DataStream API, Metrics >Reporter: Chesnay Schepler >Assignee: lining >Priority: Major > Fix For: 1.7.0 > > > To properly expose the number of records read by sources / emitted by sink we > have to expose the operator counter metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10166) Dependency problems when executing SQL query in sql-client
[
https://issues.apache.org/jira/browse/FLINK-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther updated FLINK-10166:
-
Fix Version/s: 1.5.6
> Dependency problems when executing SQL query in sql-client
> --
>
> Key: FLINK-10166
> URL: https://issues.apache.org/jira/browse/FLINK-10166
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.6.0
>Reporter: Dawid Wysakowicz
>Assignee: Timo Walther
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.5.6, 1.6.3, 1.7.0
>
>
> When tried to run query:
> {code}
> select count(distinct name) from (Values ('a'), ('b')) AS NameTable(name)
> {code}
> in {{sql-client.sh}} I got:
> {code}
> [ERROR] Could not execute SQL statement. Reason:
> org.codehaus.commons.compiler.CompileException: Line 43, Column 10: Unknown
> variable or type "org.apache.commons.codec.binary.Base64"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-9050) Expose operator IO counter metrics
[ https://issues.apache.org/jira/browse/FLINK-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669829#comment-16669829 ] lining edited comment on FLINK-9050 at 10/31/18 9:40 AM: - Hi, [~wolli]. add new rest api or add this as operator_metric in */jobs/:jobid?* was (Author: lining): Hi, [~wolli]. add new rest api for operator or add this metric to */jobs/:jobid* > Expose operator IO counter metrics > -- > > Key: FLINK-9050 > URL: https://issues.apache.org/jira/browse/FLINK-9050 > Project: Flink > Issue Type: New Feature > Components: DataSet API, DataStream API, Metrics >Reporter: Chesnay Schepler >Assignee: lining >Priority: Major > Fix For: 1.7.0 > > > To properly expose the number of records read by sources / emitted by sink we > have to expose the operator counter metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9050) Expose operator IO counter metrics
[ https://issues.apache.org/jira/browse/FLINK-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669835#comment-16669835 ] Chesnay Schepler commented on FLINK-9050: - [~lining] This issue is not about changing the REST API or WebUI in any way; it is about exposing means to sources/sink to measure incoming/outgoing data. > Expose operator IO counter metrics > -- > > Key: FLINK-9050 > URL: https://issues.apache.org/jira/browse/FLINK-9050 > Project: Flink > Issue Type: New Feature > Components: DataSet API, DataStream API, Metrics >Reporter: Chesnay Schepler >Assignee: lining >Priority: Major > Fix For: 1.7.0 > > > To properly expose the number of records read by sources / emitted by sink we > have to expose the operator counter metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (FLINK-10166) Dependency problems when executing SQL query in sql-client
[
https://issues.apache.org/jira/browse/FLINK-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther resolved FLINK-10166.
--
Resolution: Fixed
> Dependency problems when executing SQL query in sql-client
> --
>
> Key: FLINK-10166
> URL: https://issues.apache.org/jira/browse/FLINK-10166
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.6.0
>Reporter: Dawid Wysakowicz
>Assignee: Timo Walther
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.5.6, 1.6.3, 1.7.0
>
>
> When tried to run query:
> {code}
> select count(distinct name) from (Values ('a'), ('b')) AS NameTable(name)
> {code}
> in {{sql-client.sh}} I got:
> {code}
> [ERROR] Could not execute SQL statement. Reason:
> org.codehaus.commons.compiler.CompileException: Line 43, Column 10: Unknown
> variable or type "org.apache.commons.codec.binary.Base64"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10166) Dependency problems when executing SQL query in sql-client
[
https://issues.apache.org/jira/browse/FLINK-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669836#comment-16669836
]
Timo Walther commented on FLINK-10166:
--
Fixed in 1.6.3: 2657868eb704f148d5f3f1a24eaa07ad2243b189
Fixed in 1.5.6: 78525bfb2b9524a6ba02ad0b1fe09a6b0a328bdd
> Dependency problems when executing SQL query in sql-client
> --
>
> Key: FLINK-10166
> URL: https://issues.apache.org/jira/browse/FLINK-10166
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.6.0
>Reporter: Dawid Wysakowicz
>Assignee: Timo Walther
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.5.6, 1.6.3, 1.7.0
>
>
> When tried to run query:
> {code}
> select count(distinct name) from (Values ('a'), ('b')) AS NameTable(name)
> {code}
> in {{sql-client.sh}} I got:
> {code}
> [ERROR] Could not execute SQL statement. Reason:
> org.codehaus.commons.compiler.CompileException: Line 43, Column 10: Unknown
> variable or type "org.apache.commons.codec.binary.Base64"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9752) Add an S3 RecoverableWriter
[ https://issues.apache.org/jira/browse/FLINK-9752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kostas Kloudas updated FLINK-9752: -- Fix Version/s: (was: 1.6.2) > Add an S3 RecoverableWriter > --- > > Key: FLINK-9752 > URL: https://issues.apache.org/jira/browse/FLINK-9752 > Project: Flink > Issue Type: Sub-task > Components: Streaming Connectors >Reporter: Stephan Ewen >Assignee: Kostas Kloudas >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > S3 offers persistence only when uploads are complete. That means at the end > of simple uploads and uploads of parts of a MultiPartUpload. > We should implement a RecoverableWriter for S3 that does a MultiPartUpload > with a Part per checkpoint. > Recovering the reader needs the MultiPartUploadID and the list of ETags of > previous parts. > We need additional staging of data in Flink state to work around the fact that > - Parts in a MultiPartUpload must be at least 5MB > - Part sizes must be known up front. (Note that data can still be streamed > in the upload) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (FLINK-9752) Add an S3 RecoverableWriter
[ https://issues.apache.org/jira/browse/FLINK-9752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kostas Kloudas reopened FLINK-9752: --- > Add an S3 RecoverableWriter > --- > > Key: FLINK-9752 > URL: https://issues.apache.org/jira/browse/FLINK-9752 > Project: Flink > Issue Type: Sub-task > Components: Streaming Connectors >Reporter: Stephan Ewen >Assignee: Kostas Kloudas >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > S3 offers persistence only when uploads are complete. That means at the end > of simple uploads and uploads of parts of a MultiPartUpload. > We should implement a RecoverableWriter for S3 that does a MultiPartUpload > with a Part per checkpoint. > Recovering the reader needs the MultiPartUploadID and the list of ETags of > previous parts. > We need additional staging of data in Flink state to work around the fact that > - Parts in a MultiPartUpload must be at least 5MB > - Part sizes must be known up front. (Note that data can still be streamed > in the upload) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-9752) Add an S3 RecoverableWriter
[ https://issues.apache.org/jira/browse/FLINK-9752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kostas Kloudas closed FLINK-9752. - Resolution: Fixed > Add an S3 RecoverableWriter > --- > > Key: FLINK-9752 > URL: https://issues.apache.org/jira/browse/FLINK-9752 > Project: Flink > Issue Type: Sub-task > Components: Streaming Connectors >Reporter: Stephan Ewen >Assignee: Kostas Kloudas >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > S3 offers persistence only when uploads are complete. That means at the end > of simple uploads and uploads of parts of a MultiPartUpload. > We should implement a RecoverableWriter for S3 that does a MultiPartUpload > with a Part per checkpoint. > Recovering the reader needs the MultiPartUploadID and the list of ETags of > previous parts. > We need additional staging of data in Flink state to work around the fact that > - Parts in a MultiPartUpload must be at least 5MB > - Part sizes must be known up front. (Note that data can still be streamed > in the upload) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9050) Expose operator IO counter metrics
[ https://issues.apache.org/jira/browse/FLINK-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669840#comment-16669840 ] Chesnay Schepler commented on FLINK-9050: - [~lining] Please ask the currently assigned person before assigning yourself. > Expose operator IO counter metrics > -- > > Key: FLINK-9050 > URL: https://issues.apache.org/jira/browse/FLINK-9050 > Project: Flink > Issue Type: New Feature > Components: DataSet API, DataStream API, Metrics >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.7.0 > > > To properly expose the number of records read by sources / emitted by sink we > have to expose the operator counter metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229622248
##
File path: flink-end-to-end-tests/test-scripts/kafka-common.sh
##
@@ -80,13 +82,13 @@ function stop_kafka_cluster {
$KAFKA_DIR/bin/kafka-server-stop.sh
$KAFKA_DIR/bin/zookeeper-server-stop.sh
- PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
+ PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}'|| echo "")
Review comment:
Add comment `# Terminate Kafka process if it still exists`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Assigned] (FLINK-9050) Expose operator IO counter metrics
[ https://issues.apache.org/jira/browse/FLINK-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler reassigned FLINK-9050: --- Assignee: Chesnay Schepler (was: lining) > Expose operator IO counter metrics > -- > > Key: FLINK-9050 > URL: https://issues.apache.org/jira/browse/FLINK-9050 > Project: Flink > Issue Type: New Feature > Components: DataSet API, DataStream API, Metrics >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.7.0 > > > To properly expose the number of records read by sources / emitted by sink we > have to expose the operator counter metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229624262
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -60,6 +60,18 @@ under the License.
org.apache.flink
flink-connector-kafka-0.10_${scala.binary.version}
${project.version}
+
Review comment:
Why are you excluding this dependency here? end-to-end tests should be the
final validation that our code works and having to modify dependencies doesn't
sound good and might hide some error.
I wonder if maybe a better solution would be to create some separate sub
modules for examples with kafka 0.10 and kafka 2.0?
@tzulitai what do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229622276
##
File path: flink-end-to-end-tests/test-scripts/kafka-common.sh
##
@@ -80,13 +82,13 @@ function stop_kafka_cluster {
$KAFKA_DIR/bin/kafka-server-stop.sh
$KAFKA_DIR/bin/zookeeper-server-stop.sh
- PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
+ PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}'|| echo "")
if [ ! -z "$PIDS" ]; then
kill -s TERM $PIDS
fi
- PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}')
+ PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}'|| echo "")
Review comment:
`# Terminate QuorumPeerMain process if it still exists`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669841#comment-16669841
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229622276
##
File path: flink-end-to-end-tests/test-scripts/kafka-common.sh
##
@@ -80,13 +82,13 @@ function stop_kafka_cluster {
$KAFKA_DIR/bin/kafka-server-stop.sh
$KAFKA_DIR/bin/zookeeper-server-stop.sh
- PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
+ PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}'|| echo "")
if [ ! -z "$PIDS" ]; then
kill -s TERM $PIDS
fi
- PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}')
+ PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}'|| echo "")
Review comment:
`# Terminate QuorumPeerMain process if it still exists`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669843#comment-16669843
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229624262
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -60,6 +60,18 @@ under the License.
org.apache.flink
flink-connector-kafka-0.10_${scala.binary.version}
${project.version}
+
Review comment:
Why are you excluding this dependency here? end-to-end tests should be the
final validation that our code works and having to modify dependencies doesn't
sound good and might hide some error.
I wonder if maybe a better solution would be to create some separate sub
modules for examples with kafka 0.10 and kafka 2.0?
@tzulitai what do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669842#comment-16669842
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229622248
##
File path: flink-end-to-end-tests/test-scripts/kafka-common.sh
##
@@ -80,13 +82,13 @@ function stop_kafka_cluster {
$KAFKA_DIR/bin/kafka-server-stop.sh
$KAFKA_DIR/bin/zookeeper-server-stop.sh
- PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
+ PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}'|| echo "")
Review comment:
Add comment `# Terminate Kafka process if it still exists`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-9050) Expose operator IO counter metrics
[ https://issues.apache.org/jira/browse/FLINK-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669829#comment-16669829 ] lining edited comment on FLINK-9050 at 10/31/18 9:54 AM: - Add new rest api or add this as operator_metric in */jobs/:jobid?* was (Author: lining): Hi, [~wolli]. add new rest api or add this as operator_metric in */jobs/:jobid?* > Expose operator IO counter metrics > -- > > Key: FLINK-9050 > URL: https://issues.apache.org/jira/browse/FLINK-9050 > Project: Flink > Issue Type: New Feature > Components: DataSet API, DataStream API, Metrics >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.7.0 > > > To properly expose the number of records read by sources / emitted by sink we > have to expose the operator counter metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229624262
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -60,6 +60,18 @@ under the License.
org.apache.flink
flink-connector-kafka-0.10_${scala.binary.version}
${project.version}
+
Review comment:
Why are you excluding this dependency here?
This worries me, because end-to-end tests should be the final validation
that our code works and having to modify dependencies doesn't sound good and
might hide some error
I wonder if maybe a better solution would be to create some separate sub
modules for examples with kafka 0.10 and kafka 2.0?
@tzulitai what do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229624262
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -60,6 +60,18 @@ under the License.
org.apache.flink
flink-connector-kafka-0.10_${scala.binary.version}
${project.version}
+
Review comment:
Why are you excluding this dependency here?
This worries me, because end-to-end tests should be the final validation
that our code works and having to modify dependencies doesn't sound good and
might hide or cause some errors
I wonder if maybe a better solution would be to create some separate sub
modules for examples with kafka 0.10 and kafka 2.0?
@tzulitai what do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669847#comment-16669847
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229624262
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -60,6 +60,18 @@ under the License.
org.apache.flink
flink-connector-kafka-0.10_${scala.binary.version}
${project.version}
+
Review comment:
Why are you excluding this dependency here?
This worries me, because end-to-end tests should be the final validation
that our code works and having to modify dependencies doesn't sound good and
might hide or cause some errors
I wonder if maybe a better solution would be to create some separate sub
modules for examples with kafka 0.10 and kafka 2.0?
@tzulitai what do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[
https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669846#comment-16669846
]
ASF GitHub Bot commented on FLINK-10600:
pnowojski commented on a change in pull request #6924: [FLINK-10600] Provide
End-to-end test cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#discussion_r229624262
##
File path: flink-examples/flink-examples-streaming/pom.xml
##
@@ -60,6 +60,18 @@ under the License.
org.apache.flink
flink-connector-kafka-0.10_${scala.binary.version}
${project.version}
+
Review comment:
Why are you excluding this dependency here?
This worries me, because end-to-end tests should be the final validation
that our code works and having to modify dependencies doesn't sound good and
might hide some error
I wonder if maybe a better solution would be to create some separate sub
modules for examples with kafka 0.10 and kafka 2.0?
@tzulitai what do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide End-to-end test cases for modern Kafka connectors
> -
>
> Key: FLINK-10600
> URL: https://issues.apache.org/jira/browse/FLINK-10600
> Project: Flink
> Issue Type: Sub-task
> Components: Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] pnowojski commented on a change in pull request #6963: [FLINK-10704] Fix sql client end to end test failure
pnowojski commented on a change in pull request #6963: [FLINK-10704] Fix sql
client end to end test failure
URL: https://github.com/apache/flink/pull/6963#discussion_r229627265
##
File path: flink-end-to-end-tests/test-scripts/common.sh
##
@@ -316,7 +316,7 @@ function check_logs_for_errors {
| grep -v "java.lang.NoClassDefFoundError:
org/apache/hadoop/yarn/exceptions/YarnException" \
| grep -v "java.lang.NoClassDefFoundError:
org/apache/hadoop/conf/Configuration" \
| grep -v
"org.apache.flink.fs.shaded.hadoop3.org.apache.commons.beanutils.FluentPropertyBeanIntrospector
- Error when creating PropertyDescriptor for public final void
org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
Ignoring this property." \
- | grep -ic "error")
+ | grep -c "ERROR")
Review comment:
Initially I though about ignoring all warnings, since it seemed to me as a
better solution compared to changing the behaviour of case sensitiveness
globally. But Then I realised that there are already way too many exclusions
hardcoded in this file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-10732) force-shading 1.5.5 maven artifact was not released
[
https://issues.apache.org/jira/browse/FLINK-10732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669852#comment-16669852
]
Maximilian Michels commented on FLINK-10732:
I've also ran into this and worked around by excluding it from the Flink
dependencies. I haven't seen any other artifacts missing.
Could we redeploy the force-shading?
> force-shading 1.5.5 maven artifact was not released
> ---
>
> Key: FLINK-10732
> URL: https://issues.apache.org/jira/browse/FLINK-10732
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.5.5
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Fix For: 1.5.5
>
>
> The 1.5.5 maven artifact for {{force-shading}} was not deployed. We have to
> investigate whether other artifacts are missing as well and how to remedy the
> problem, i.e. re-deploy the missing artifacts (which may or may not require a
> separate vote).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10704) Fix sql client end to end test failure
[
https://issues.apache.org/jira/browse/FLINK-10704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669851#comment-16669851
]
ASF GitHub Bot commented on FLINK-10704:
pnowojski commented on a change in pull request #6963: [FLINK-10704] Fix sql
client end to end test failure
URL: https://github.com/apache/flink/pull/6963#discussion_r229627265
##
File path: flink-end-to-end-tests/test-scripts/common.sh
##
@@ -316,7 +316,7 @@ function check_logs_for_errors {
| grep -v "java.lang.NoClassDefFoundError:
org/apache/hadoop/yarn/exceptions/YarnException" \
| grep -v "java.lang.NoClassDefFoundError:
org/apache/hadoop/conf/Configuration" \
| grep -v
"org.apache.flink.fs.shaded.hadoop3.org.apache.commons.beanutils.FluentPropertyBeanIntrospector
- Error when creating PropertyDescriptor for public final void
org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
Ignoring this property." \
- | grep -ic "error")
+ | grep -c "ERROR")
Review comment:
Initially I though about ignoring all warnings, since it seemed to me as a
better solution compared to changing the behaviour of case sensitiveness
globally. But Then I realised that there are already way too many exclusions
hardcoded in this file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fix sql client end to end test failure
> --
>
> Key: FLINK-10704
> URL: https://issues.apache.org/jira/browse/FLINK-10704
> Project: Flink
> Issue Type: Bug
> Components: E2E Tests, Kafka Connector
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
>
> The log file contains the following sentence:
> {code:java}
> 2018-10-29 03:27:39,209 WARN
> org.apache.flink.kafka010.shaded.org.apache.kafka.common.utils.AppInfoParser
> - Error while loading kafka-version.properties :null
> {code}
> The reason for this log is that we explicitly exclude the version description
> file of the kafka client when packaging the connector:
> {code:java}
>
>
> *:*
>
> kafka/kafka-version.properties
>
>
> {code}
> When the shell scan the "error" keyword with grep, it will hit, so the test
> will fail.
> {code:java}
> function check_logs_for_errors {
> error_count=$(grep -rv "GroupCoordinatorNotAvailableException"
> $FLINK_DIR/log \
> | grep -v "RetriableCommitFailedException" \
> | grep -v "NoAvailableBrokersException" \
> | grep -v "Async Kafka commit failed" \
> | grep -v "DisconnectException" \
> | grep -v "AskTimeoutException" \
> | grep -v "WARN akka.remote.transport.netty.NettyTransport" \
> | grep -v "WARN
> org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline" \
> | grep -v "jvm-exit-on-fatal-error" \
> | grep -v '^INFO:.*AWSErrorCode=\[400 Bad
> Request\].*ServiceEndpoint=\[https://.*\.s3\.amazonaws\.com\].*RequestType=\[HeadBucketRequest\]'
> \
> | grep -v "RejectedExecutionException" \
> | grep -v "An exception was thrown by an exception handler" \
> | grep -v "java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/exceptions/YarnException" \
> | grep -v "java.lang.NoClassDefFoundError:
> org/apache/hadoop/conf/Configuration" \
> | grep -v
> "org.apache.flink.fs.shaded.hadoop3.org.apache.commons.beanutils.FluentPropertyBeanIntrospector
> - Error when creating PropertyDescriptor for public final void
> org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)!
> Ignoring this property." \
> | grep -ic "error")//here
> if [[ ${error_count} -gt 0 ]]; then
> echo "Found error in log files:"
> cat $FLINK_DIR/log/*
> EXIT_CODE=1
> fi
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] tillrohrmann commented on issue #6684: [FLINK-10205] Batch Job: InputSplit Fault tolerant for DataSource…
tillrohrmann commented on issue #6684: [FLINK-10205] Batch Job: InputSplit Fault tolerant for DataSource… URL: https://github.com/apache/flink/pull/6684#issuecomment-434628107 Thanks for the explanation @isunjin. Now I understand that you don't think that it is strictly required to let a failed task process exactly the same `InputSplits` and that it is just a side effect of the current implementation. So in the end you've implemented it this way, because the `InputSplitAssigner` does not support returning `InputSplits`. Maybe that is something we should change. We could, for example, add a new interface which needs to be implemented by an `InputSplitAssigner` to support fine grained recovery. Otherwise, such a failure will result into a global failover. My concern is that by storing `InputSplits` in the `Executions` that we are mixing a bit of concerns. For example, assume that we have three tasks failing and we also lost a slot. Then we could only restart two of these tasks and need to distribute the slots of the third `Execution` among the newly started `Executions`. It would be much easier to simply return all slots to the `InputSplitAssigner` and let the newly started `Executions` pull from there. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zhangxinyu1 commented on issue #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly.
zhangxinyu1 commented on issue #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#issuecomment-434628064 > I think we missed an important thing. We should offer file size to user. If not, the parameters 'start', 'end' or 'size' would be useless since the user may have no idea how to set these parameters. Maybe we can set the start=0 and size=fileSize as the default value when user choose a log file. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669857#comment-16669857 ] ASF GitHub Bot commented on FLINK-10002: zhangxinyu1 commented on issue #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770#issuecomment-434628064 > I think we missed an important thing. We should offer file size to user. If not, the parameters 'start', 'end' or 'size' would be useless since the user may have no idea how to set these parameters. Maybe we can set the start=0 and size=fileSize as the default value when user choose a log file. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10002) WebUI shows logs unfriendly, especially when the amount of logs is large
[ https://issues.apache.org/jira/browse/FLINK-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669860#comment-16669860 ] ASF GitHub Bot commented on FLINK-10002: zhangxinyu1 opened a new pull request #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770 ## What is the purpose of the change This commit makes WebUI show jm/tm logs more friendly. WebUI can show historical logs by inputing a filename and a offset range to read of the file, shown as follows.     This pr adds 6 new REST APIs: - For JobManager `/jobmanager/log?filename=xxx&start=0&size=10240`Return logs of the current log file or a given file in the range. `/jobmanager/loglist`Return the list of historical log filename. `/jobmanager/stdout?start=0&size=10240`Return the stdout data in the range - For TaskManager `/taskmanagers/:taskmanagerid/log?filename=xxx&start=0&size=10240`Return logs of the current log file or a given file in the range. `/taskmanagers/:taskmanagerid/loglist`Return the list of historical log filename. `/taskmanagers/:taskmanagerid/stdout?start=0&size=10240`Return the stdout data in the range ## Brief change log This pr mainly add 6 APIs in new (WebMonitorEndpoint) REST API. Besides, some related frontend codes are changed, such as 1. `JobManagerLogFileHandler` is used to handle the request to fetch stdout and log file of job manager. The related headers are `JobManagerLogFileHeaders` and `JobManagerStdoutFileHeaders`. 2. `JobManagerLogListHandler` is used to handle the request to fetch log list of job manager. The related header is `JobManagerLogListHeaders`, and return a `LogListInfo` value. 3. `TaskManagerLogHandler` is used to handle the request to fetch stdout and log file of a given task manager. The related headers are `TaskManagerLogFileHeaders` and `TaskManagerStdoutFileHeaders`. 4. `TaskManagerLogListHandler` is used to handle the request to fetch log list of a given task manager. The related header is `TaskManagerLogListHeaders`. 5. `FileOffsetRange` is designed to identify a range of log files, which includes a start offset and an end offset. `LogFilenameQueryParameter`, `LogSizeQueryParameter` and `LogStartOffsetQueryParameter` are the parameters of requests. ## Verifying this change No tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? JavaDocs This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > WebUI shows logs unfriendly, especially when the amount of logs is large > > > Key: FLINK-10002 > URL: https://issues.apache.org/jira/browse/FLINK-10002 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: zhangxinyu >Assignee: zhangxinyu >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Attachments: image-2018-09-10-11-38-07-973.png > > > When a streaming job run for a long time, the amount of logs may be very > large. The current WebUI shows all content of logs. It will cost much time to > download logs from task managers. and the browser cannot display the logs. > Therefore, I suggest that Flink uses DailyRollingAppender to split logs by > default, and task manager provides an API that can get logs based on a > parameter of time interval. In this way WebUI can display logs based on time > interval. -- This message was sent by Atlassian JIRA
[jira] [Commented] (FLINK-10205) Batch Job: InputSplit Fault tolerant for DataSourceTask
[ https://issues.apache.org/jira/browse/FLINK-10205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669861#comment-16669861 ] ASF GitHub Bot commented on FLINK-10205: tillrohrmann commented on issue #6684: [FLINK-10205] Batch Job: InputSplit Fault tolerant for DataSource… URL: https://github.com/apache/flink/pull/6684#issuecomment-434628107 Thanks for the explanation @isunjin. Now I understand that you don't think that it is strictly required to let a failed task process exactly the same `InputSplits` and that it is just a side effect of the current implementation. So in the end you've implemented it this way, because the `InputSplitAssigner` does not support returning `InputSplits`. Maybe that is something we should change. We could, for example, add a new interface which needs to be implemented by an `InputSplitAssigner` to support fine grained recovery. Otherwise, such a failure will result into a global failover. My concern is that by storing `InputSplits` in the `Executions` that we are mixing a bit of concerns. For example, assume that we have three tasks failing and we also lost a slot. Then we could only restart two of these tasks and need to distribute the slots of the third `Execution` among the newly started `Executions`. It would be much easier to simply return all slots to the `InputSplitAssigner` and let the newly started `Executions` pull from there. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Batch Job: InputSplit Fault tolerant for DataSourceTask > --- > > Key: FLINK-10205 > URL: https://issues.apache.org/jira/browse/FLINK-10205 > Project: Flink > Issue Type: Sub-task > Components: JobManager >Affects Versions: 1.6.1, 1.6.2, 1.7.0 >Reporter: JIN SUN >Assignee: JIN SUN >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Today DataSource Task pull InputSplits from JobManager to achieve better > performance, however, when a DataSourceTask failed and rerun, it will not get > the same splits as its previous version. this will introduce inconsistent > result or even data corruption. > Furthermore, if there are two executions run at the same time (in batch > scenario), this two executions should process same splits. > we need to fix the issue to make the inputs of a DataSourceTask > deterministic. The propose is save all splits into ExecutionVertex and > DataSourceTask will pull split from there. > document: > [https://docs.google.com/document/d/1FdZdcA63tPUEewcCimTFy9Iz2jlVlMRANZkO4RngIuk/edit?usp=sharing] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] zhangxinyu1 opened a new pull request #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly.
zhangxinyu1 opened a new pull request #6770: [FLINK-10002] [Webfrontend] WebUI shows jm/tm logs more friendly. URL: https://github.com/apache/flink/pull/6770 ## What is the purpose of the change This commit makes WebUI show jm/tm logs more friendly. WebUI can show historical logs by inputing a filename and a offset range to read of the file, shown as follows.     This pr adds 6 new REST APIs: - For JobManager `/jobmanager/log?filename=xxx&start=0&size=10240`Return logs of the current log file or a given file in the range. `/jobmanager/loglist`Return the list of historical log filename. `/jobmanager/stdout?start=0&size=10240`Return the stdout data in the range - For TaskManager `/taskmanagers/:taskmanagerid/log?filename=xxx&start=0&size=10240`Return logs of the current log file or a given file in the range. `/taskmanagers/:taskmanagerid/loglist`Return the list of historical log filename. `/taskmanagers/:taskmanagerid/stdout?start=0&size=10240`Return the stdout data in the range ## Brief change log This pr mainly add 6 APIs in new (WebMonitorEndpoint) REST API. Besides, some related frontend codes are changed, such as 1. `JobManagerLogFileHandler` is used to handle the request to fetch stdout and log file of job manager. The related headers are `JobManagerLogFileHeaders` and `JobManagerStdoutFileHeaders`. 2. `JobManagerLogListHandler` is used to handle the request to fetch log list of job manager. The related header is `JobManagerLogListHeaders`, and return a `LogListInfo` value. 3. `TaskManagerLogHandler` is used to handle the request to fetch stdout and log file of a given task manager. The related headers are `TaskManagerLogFileHeaders` and `TaskManagerStdoutFileHeaders`. 4. `TaskManagerLogListHandler` is used to handle the request to fetch log list of a given task manager. The related header is `TaskManagerLogListHeaders`. 5. `FileOffsetRange` is designed to identify a range of log files, which includes a start offset and an end offset. `LogFilenameQueryParameter`, `LogSizeQueryParameter` and `LogStartOffsetQueryParameter` are the parameters of requests. ## Verifying this change No tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? JavaDocs This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10732) force-shading 1.5.5 maven artifact was not released
[
https://issues.apache.org/jira/browse/FLINK-10732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669862#comment-16669862
]
Chesnay Schepler commented on FLINK-10732:
--
Currently looking into how/whether we can simply re-deploy the artifact through
nexus.
> force-shading 1.5.5 maven artifact was not released
> ---
>
> Key: FLINK-10732
> URL: https://issues.apache.org/jira/browse/FLINK-10732
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.5.5
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Fix For: 1.5.5
>
>
> The 1.5.5 maven artifact for {{force-shading}} was not deployed. We have to
> investigate whether other artifacts are missing as well and how to remedy the
> problem, i.e. re-deploy the missing artifacts (which may or may not require a
> separate vote).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9050) Expose operator IO counter metrics
[ https://issues.apache.org/jira/browse/FLINK-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669863#comment-16669863 ] lining commented on FLINK-9050: --- [~Zentol] Sorry. > Expose operator IO counter metrics > -- > > Key: FLINK-9050 > URL: https://issues.apache.org/jira/browse/FLINK-9050 > Project: Flink > Issue Type: New Feature > Components: DataSet API, DataStream API, Metrics >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.7.0 > > > To properly expose the number of records read by sources / emitted by sink we > have to expose the operator counter metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (FLINK-9635) Local recovery scheduling can cause spread out of tasks
[
https://issues.apache.org/jira/browse/FLINK-9635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann reopened FLINK-9635:
--
> Local recovery scheduling can cause spread out of tasks
> ---
>
> Key: FLINK-9635
> URL: https://issues.apache.org/jira/browse/FLINK-9635
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination
>Affects Versions: 1.5.0, 1.6.2
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> In order to make local recovery work, Flink's scheduling was changed such
> that it tries to be rescheduled to its previous location. In order to not
> occupy slots which have state of other tasks cached, the strategy will
> request a new slot if the old slot identified by the previous allocation id
> is no longer present. This also applies to newly allocated slots because
> there is no distinction between new or already used. This behaviour can cause
> that every tasks gets deployed to its own slot if the {{SlotPool}} has
> released all slots in the meantime, for example. The consequence could be
> that a job can no longer be executed after a failure because it needs more
> slots than before.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] StefanRRichter commented on issue #6684: [FLINK-10205] Batch Job: InputSplit Fault tolerant for DataSource…
StefanRRichter commented on issue #6684: [FLINK-10205] Batch Job: InputSplit Fault tolerant for DataSource… URL: https://github.com/apache/flink/pull/6684#issuecomment-434629432 +1 for exploring a way to return `InputSplits` to the `InputSplitAssigner`. The mix of concerns within `Executions` was also one of my main concerns, would be good if we find a way to avoid that. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-9635) Local recovery scheduling can cause spread out of tasks
[
https://issues.apache.org/jira/browse/FLINK-9635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-9635.
Resolution: Fixed
Release Note: With the improvements to Flink's scheduling, it can no longer
happen that recoveries require more slots than before if local recovery is
enabled. Consequently, we encourage our users to use the local recovery feature
which can be enabled by `state.backend.local-recovery: true`.
> Local recovery scheduling can cause spread out of tasks
> ---
>
> Key: FLINK-9635
> URL: https://issues.apache.org/jira/browse/FLINK-9635
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination
>Affects Versions: 1.5.0, 1.6.2
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> In order to make local recovery work, Flink's scheduling was changed such
> that it tries to be rescheduled to its previous location. In order to not
> occupy slots which have state of other tasks cached, the strategy will
> request a new slot if the old slot identified by the previous allocation id
> is no longer present. This also applies to newly allocated slots because
> there is no distinction between new or already used. This behaviour can cause
> that every tasks gets deployed to its own slot if the {{SlotPool}} has
> released all slots in the meantime, for example. The consequence could be
> that a job can no longer be executed after a failure because it needs more
> slots than before.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10205) Batch Job: InputSplit Fault tolerant for DataSourceTask
[ https://issues.apache.org/jira/browse/FLINK-10205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669867#comment-16669867 ] ASF GitHub Bot commented on FLINK-10205: StefanRRichter commented on issue #6684: [FLINK-10205] Batch Job: InputSplit Fault tolerant for DataSource… URL: https://github.com/apache/flink/pull/6684#issuecomment-434629432 +1 for exploring a way to return `InputSplits` to the `InputSplitAssigner`. The mix of concerns within `Executions` was also one of my main concerns, would be good if we find a way to avoid that. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Batch Job: InputSplit Fault tolerant for DataSourceTask > --- > > Key: FLINK-10205 > URL: https://issues.apache.org/jira/browse/FLINK-10205 > Project: Flink > Issue Type: Sub-task > Components: JobManager >Affects Versions: 1.6.1, 1.6.2, 1.7.0 >Reporter: JIN SUN >Assignee: JIN SUN >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Today DataSource Task pull InputSplits from JobManager to achieve better > performance, however, when a DataSourceTask failed and rerun, it will not get > the same splits as its previous version. this will introduce inconsistent > result or even data corruption. > Furthermore, if there are two executions run at the same time (in batch > scenario), this two executions should process same splits. > we need to fix the issue to make the inputs of a DataSourceTask > deterministic. The propose is save all splits into ExecutionVertex and > DataSourceTask will pull split from there. > document: > [https://docs.google.com/document/d/1FdZdcA63tPUEewcCimTFy9Iz2jlVlMRANZkO4RngIuk/edit?usp=sharing] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10731) Support AVG on Date fields
[
https://issues.apache.org/jira/browse/FLINK-10731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669871#comment-16669871
]
Flavio Pompermaier commented on FLINK-10731:
{code:java}
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
BatchTableEnvironment btEnv = TableEnvironment.getTableEnvironment(env);
btEnv.registerDataSet("TEST_TABLE", env.fromCollection(Arrays.asList(new
Date())), "somedate");
Table table = btEnv.sqlQuery("SELECT AVG(somedate) FROM TEST_TABLE");
btEnv.toDataSet(table, new RowTypeInfo(BasicTypeInfo.DATE_TYPE_INFO)).print();
{code}
> Support AVG on Date fields
> --
>
> Key: FLINK-10731
> URL: https://issues.apache.org/jira/browse/FLINK-10731
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.6.2
>Reporter: Flavio Pompermaier
>Priority: Minor
>
> AVG function does not work on date fields right now
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] tillrohrmann commented on issue #6951: [FLINK-10693] [Scala API] Fix incorrect duplication in EitherSerializer
tillrohrmann commented on issue #6951: [FLINK-10693] [Scala API] Fix incorrect duplication in EitherSerializer URL: https://github.com/apache/flink/pull/6951#issuecomment-434631198 I'll backport this PR to `release-1.5` and `release-1.6`. If Travis passes, then I'll merge it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10693) Fix Scala EitherSerializer duplication
[ https://issues.apache.org/jira/browse/FLINK-10693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669874#comment-16669874 ] ASF GitHub Bot commented on FLINK-10693: tillrohrmann commented on issue #6951: [FLINK-10693] [Scala API] Fix incorrect duplication in EitherSerializer URL: https://github.com/apache/flink/pull/6951#issuecomment-434631198 I'll backport this PR to `release-1.5` and `release-1.6`. If Travis passes, then I'll merge it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix Scala EitherSerializer duplication > -- > > Key: FLINK-10693 > URL: https://issues.apache.org/jira/browse/FLINK-10693 > Project: Flink > Issue Type: Bug > Components: Scala API >Affects Versions: 1.6.1 >Reporter: Stephan Ewen >Assignee: Stephan Ewen >Priority: Critical > Labels: pull-request-available > Fix For: 1.7.0 > > > The Scala Either Serializer has buggy duplication logic, resulting in sharing > and incorrect concurrent use when the nested serializers are not thread safe. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] dawidwys commented on a change in pull request #6943: [FLINK-10632][e2e] Running general purpose testing job with failure in per-job mode
dawidwys commented on a change in pull request #6943: [FLINK-10632][e2e]
Running general purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#discussion_r229632726
##
File path:
flink-end-to-end-tests/test-scripts/test_ha_per_job_cluster_datastream.sh
##
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_ha.sh
+
+TEST_PROGRAM_JAR_NAME=DataStreamAllroundTestProgram.jar
+TEST_PROGRAM_JAR=${END_TO_END_DIR}/flink-datastream-allround-test/target/${TEST_PROGRAM_JAR_NAME}
+FLINK_LIB_DIR=${FLINK_DIR}/lib
+JOB_ID=""
+
+function ha_cleanup() {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ stop_watchdogs
+ kill_all 'StandaloneJobClusterEntryPoint'
+ shutdown_all
+ rm ${FLINK_LIB_DIR}/${TEST_PROGRAM_JAR_NAME}
+}
+
+trap ha_cleanup INT
+trap ha_cleanup EXIT
+
+function run_job() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local CHECKPOINT_DIR="${TEST_DATA_DIR}/checkpoints/"
+
+${FLINK_DIR}/bin/standalone-job.sh start \
+--job-classname
org.apache.flink.streaming.tests.DataStreamAllroundTestProgram \
+-p ${PARALLELISM} \
+--environment.parallelism ${PARALLELISM} \
+--test.semantics exactly-once \
+--test.simulate_failure true \
+--test.simulate_failure.num_records 200 \
+--test.simulate_failure.num_checkpoints 1 \
+--test.simulate_failure.max_failures 20 \
+--state_backend ${BACKEND} \
+--state_backend.checkpoint_directory "file://${CHECKPOINT_DIR}" \
+--state_backend.file.async ${ASYNC} \
+--state_backend.rocks.incremental ${INCREM} \
+--sequence_generator_source.sleep_time 15 \
+--sequence_generator_source.sleep_after_elements 1
+}
+
+function verify_logs_per_job() {
+local OUTPUT=$FLINK_DIR/log/*.out
+local JM_FAILURES=$1
+local EXIT_CODE=0
+local VERIFY_CHECKPOINTS=$2
+
+# verify that we have no alerts
+if ! [ `cat ${OUTPUT} | wc -l` -eq 0 ]; then
+echo "FAILURE: Alerts found at the general purpose job."
+EXIT_CODE=1
+fi
+
+# checks that all apart from the first JM recover the failed jobgraph.
+if ! [ `grep -r --include '*standalonejob*.log' 'Found 0 checkpoints in
ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq 1 ]; then
+echo "FAILURE: A JM did not take over, but started new job."
+EXIT_CODE=1
+fi
+
+if ! [ `grep -r --include '*standalonejob*.log' -P 'Found \d+ checkpoints
in ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq
$((JM_FAILURES + 1)) ]; then
+echo "FAILURE: A JM did not take over."
+EXIT_CODE=1
+fi
+
+if [ "$VERIFY_CHECKPOINTS" = true ]; then
+# search the logs for JMs that log completed checkpoints
+if ! [ `grep -r --include '*standalonejob*.log' 'Completed checkpoint'
"${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq $((JM_FAILURES + 1))
]; then
+echo "FAILURE: A JM did not execute the job."
+EXIT_CODE=1
+fi
+fi
+
+if [[ $EXIT_CODE != 0 ]]; then
+echo "One or more tests FAILED."
+exit $EXIT_CODE
+fi
+}
Review comment:
It is similar, but it performs different checks. Looks for different texts
and number of them. I extracted the comparing number of occurrence logic but
don't think there is much more that can be unified.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-4582) Allow FlinkKinesisConsumer to adapt for AWS DynamoDB Streams
[
https://issues.apache.org/jira/browse/FLINK-4582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669881#comment-16669881
]
ASF GitHub Bot commented on FLINK-4582:
---
zentol commented on issue #6968: [FLINK-4582] Consuming data from DynamoDB
streams to flink
URL: https://github.com/apache/flink/pull/6968#issuecomment-434632762
@yxu-valleytider Please make sure that the Pull Request title references the
corresponding JIRA. I have modified the title accordingly this time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow FlinkKinesisConsumer to adapt for AWS DynamoDB Streams
>
>
> Key: FLINK-4582
> URL: https://issues.apache.org/jira/browse/FLINK-4582
> Project: Flink
> Issue Type: New Feature
> Components: Kinesis Connector, Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Ying Xu
>Priority: Major
> Labels: pull-request-available
>
> AWS DynamoDB is a NoSQL database service that has a CDC-like (change data
> capture) feature called DynamoDB Streams
> (http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html),
> which is a stream feed of item-level table activities.
> The DynamoDB Streams shard abstraction follows that of Kinesis Streams with
> only a slight difference in resharding behaviours, so it is possible to build
> on the internals of our Flink Kinesis Consumer for an exactly-once DynamoDB
> Streams source.
> I propose an API something like this:
> {code}
> DataStream dynamoItemsCdc =
> FlinkKinesisConsumer.asDynamoDBStream(tableNames, schema, config)
> {code}
> The feature adds more connectivity to popular AWS services for Flink, and
> combining what Flink has for exactly-once semantics, out-of-core state
> backends, and queryable state with CDC can have very strong use cases. For
> this feature there should only be an extra dependency to the AWS Java SDK for
> DynamoDB, which has Apache License 2.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zentol commented on issue #6968: [FLINK-4582] Consuming data from DynamoDB streams to flink
zentol commented on issue #6968: [FLINK-4582] Consuming data from DynamoDB streams to flink URL: https://github.com/apache/flink/pull/6968#issuecomment-434632762 @yxu-valleytider Please make sure that the Pull Request title references the corresponding JIRA. I have modified the title accordingly this time. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10632) Run general purpose test job with failures in per-job mode
[
https://issues.apache.org/jira/browse/FLINK-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669880#comment-16669880
]
ASF GitHub Bot commented on FLINK-10632:
dawidwys commented on a change in pull request #6943: [FLINK-10632][e2e]
Running general purpose testing job with failure in per-job mode
URL: https://github.com/apache/flink/pull/6943#discussion_r229632726
##
File path:
flink-end-to-end-tests/test-scripts/test_ha_per_job_cluster_datastream.sh
##
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+source "$(dirname "$0")"/common.sh
+source "$(dirname "$0")"/common_ha.sh
+
+TEST_PROGRAM_JAR_NAME=DataStreamAllroundTestProgram.jar
+TEST_PROGRAM_JAR=${END_TO_END_DIR}/flink-datastream-allround-test/target/${TEST_PROGRAM_JAR_NAME}
+FLINK_LIB_DIR=${FLINK_DIR}/lib
+JOB_ID=""
+
+function ha_cleanup() {
+ # don't call ourselves again for another signal interruption
+ trap "exit -1" INT
+ # don't call ourselves again for normal exit
+ trap "" EXIT
+
+ stop_watchdogs
+ kill_all 'StandaloneJobClusterEntryPoint'
+ shutdown_all
+ rm ${FLINK_LIB_DIR}/${TEST_PROGRAM_JAR_NAME}
+}
+
+trap ha_cleanup INT
+trap ha_cleanup EXIT
+
+function run_job() {
+local PARALLELISM=$1
+local BACKEND=$2
+local ASYNC=$3
+local INCREM=$4
+
+local CHECKPOINT_DIR="${TEST_DATA_DIR}/checkpoints/"
+
+${FLINK_DIR}/bin/standalone-job.sh start \
+--job-classname
org.apache.flink.streaming.tests.DataStreamAllroundTestProgram \
+-p ${PARALLELISM} \
+--environment.parallelism ${PARALLELISM} \
+--test.semantics exactly-once \
+--test.simulate_failure true \
+--test.simulate_failure.num_records 200 \
+--test.simulate_failure.num_checkpoints 1 \
+--test.simulate_failure.max_failures 20 \
+--state_backend ${BACKEND} \
+--state_backend.checkpoint_directory "file://${CHECKPOINT_DIR}" \
+--state_backend.file.async ${ASYNC} \
+--state_backend.rocks.incremental ${INCREM} \
+--sequence_generator_source.sleep_time 15 \
+--sequence_generator_source.sleep_after_elements 1
+}
+
+function verify_logs_per_job() {
+local OUTPUT=$FLINK_DIR/log/*.out
+local JM_FAILURES=$1
+local EXIT_CODE=0
+local VERIFY_CHECKPOINTS=$2
+
+# verify that we have no alerts
+if ! [ `cat ${OUTPUT} | wc -l` -eq 0 ]; then
+echo "FAILURE: Alerts found at the general purpose job."
+EXIT_CODE=1
+fi
+
+# checks that all apart from the first JM recover the failed jobgraph.
+if ! [ `grep -r --include '*standalonejob*.log' 'Found 0 checkpoints in
ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq 1 ]; then
+echo "FAILURE: A JM did not take over, but started new job."
+EXIT_CODE=1
+fi
+
+if ! [ `grep -r --include '*standalonejob*.log' -P 'Found \d+ checkpoints
in ZooKeeper' "${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq
$((JM_FAILURES + 1)) ]; then
+echo "FAILURE: A JM did not take over."
+EXIT_CODE=1
+fi
+
+if [ "$VERIFY_CHECKPOINTS" = true ]; then
+# search the logs for JMs that log completed checkpoints
+if ! [ `grep -r --include '*standalonejob*.log' 'Completed checkpoint'
"${FLINK_DIR}/log/" | cut -d ":" -f 1 | uniq | wc -l` -eq $((JM_FAILURES + 1))
]; then
+echo "FAILURE: A JM did not execute the job."
+EXIT_CODE=1
+fi
+fi
+
+if [[ $EXIT_CODE != 0 ]]; then
+echo "One or more tests FAILED."
+exit $EXIT_CODE
+fi
+}
Review comment:
It is similar, but it performs different checks. Looks for different texts
and number of them. I extracted the comparing number of occurrence logic but
don't think there is much more that can be unified.
This is an automated message from the Apache Git Servic
[GitHub] zentol closed pull request #6939: [FLINK-10690][tests] Fix Files.list resource leaks
zentol closed pull request #6939: [FLINK-10690][tests] Fix Files.list resource
leaks
URL: https://github.com/apache/flink/pull/6939
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/flink-core/src/test/java/org/apache/flink/util/FileUtilsTest.java
b/flink-core/src/test/java/org/apache/flink/util/FileUtilsTest.java
index 23878cbccb8..76b78056440 100644
--- a/flink-core/src/test/java/org/apache/flink/util/FileUtilsTest.java
+++ b/flink-core/src/test/java/org/apache/flink/util/FileUtilsTest.java
@@ -38,6 +38,7 @@
import java.util.Comparator;
import java.util.List;
import java.util.stream.Collectors;
+import java.util.stream.Stream;
import static org.junit.Assert.assertArrayEquals;
import static org.junit.Assert.assertEquals;
@@ -227,12 +228,18 @@ private static void assertDirEquals(java.nio.file.Path
expected, java.nio.file.P
assertEquals(expected.getFileName(), actual.getFileName());
if (Files.isDirectory(expected)) {
- List expectedContents =
Files.list(expected)
-
.sorted(Comparator.comparing(java.nio.file.Path::toString))
- .collect(Collectors.toList());
- List actualContents =
Files.list(actual)
-
.sorted(Comparator.comparing(java.nio.file.Path::toString))
- .collect(Collectors.toList());
+ List expectedContents;
+ try (Stream files =
Files.list(expected)) {
+ expectedContents = files
+
.sorted(Comparator.comparing(java.nio.file.Path::toString))
+ .collect(Collectors.toList());
+ }
+ List actualContents;
+ try (Stream files =
Files.list(actual)) {
+ actualContents = files
+
.sorted(Comparator.comparing(java.nio.file.Path::toString))
+ .collect(Collectors.toList());
+ }
assertEquals(expectedContents.size(),
actualContents.size());
diff --git
a/flink-end-to-end-tests/flink-distributed-cache-via-blob-test/src/main/java/org/apache/flink/streaming/tests/DistributedCacheViaBlobTestProgram.java
b/flink-end-to-end-tests/flink-distributed-cache-via-blob-test/src/main/java/org/apache/flink/streaming/tests/DistributedCacheViaBlobTestProgram.java
index 167101cc9aa..388cdc6adfa 100644
---
a/flink-end-to-end-tests/flink-distributed-cache-via-blob-test/src/main/java/org/apache/flink/streaming/tests/DistributedCacheViaBlobTestProgram.java
+++
b/flink-end-to-end-tests/flink-distributed-cache-via-blob-test/src/main/java/org/apache/flink/streaming/tests/DistributedCacheViaBlobTestProgram.java
@@ -26,6 +26,7 @@
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.stream.Collectors;
+import java.util.stream.Stream;
/**
* End-to-end test program for verifying that files are distributed via
BlobServer and later accessible through
@@ -47,8 +48,10 @@ public static void main(String[] args) throws Exception {
env.registerCachedFile(inputFile.toString(), "test_data",
false);
env.registerCachedFile(inputDir.toString(), "test_dir", false);
- Path containedFile = Files.list(inputDir).findAny()
- .orElseThrow(() -> new RuntimeException("Input
directory must not be empty."));
+ final Path containedFile;
+ try (Stream files = Files.list(inputDir)) {
+ containedFile = files.findAny().orElseThrow(() -> new
RuntimeException("Input directory must not be empty."));
+ }
env.fromElements(1)
.map(new TestMapFunction(
@@ -96,8 +99,10 @@ public String map(Integer value) throws Exception {
"initial dir. Input dir path: %s. Cache
dir path: %s", initialDirPath, testDir));
}
- if
(Files.list(testDir).map(Path::getFileName).map(Path::toString).noneMatch(path
-> path.equals(containedFileName))) {
- throw new
RuntimeException(String.format("Cached directory %s should not be empty.",
testDir));
+ try (Stream files = Files.list(testDir)) {
+ if
(files.map(Path::getFileName).map(Path::toString).noneMatch(path ->
path.equals(containedFileName))) {
+ throw new
RuntimeException(String.format("Cached directory %s should not be e
[jira] [Updated] (FLINK-4582) Allow FlinkKinesisConsumer to adapt for AWS DynamoDB Streams
[
https://issues.apache.org/jira/browse/FLINK-4582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-4582:
--
Labels: pull-request-available (was: )
> Allow FlinkKinesisConsumer to adapt for AWS DynamoDB Streams
>
>
> Key: FLINK-4582
> URL: https://issues.apache.org/jira/browse/FLINK-4582
> Project: Flink
> Issue Type: New Feature
> Components: Kinesis Connector, Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Ying Xu
>Priority: Major
> Labels: pull-request-available
>
> AWS DynamoDB is a NoSQL database service that has a CDC-like (change data
> capture) feature called DynamoDB Streams
> (http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html),
> which is a stream feed of item-level table activities.
> The DynamoDB Streams shard abstraction follows that of Kinesis Streams with
> only a slight difference in resharding behaviours, so it is possible to build
> on the internals of our Flink Kinesis Consumer for an exactly-once DynamoDB
> Streams source.
> I propose an API something like this:
> {code}
> DataStream dynamoItemsCdc =
> FlinkKinesisConsumer.asDynamoDBStream(tableNames, schema, config)
> {code}
> The feature adds more connectivity to popular AWS services for Flink, and
> combining what Flink has for exactly-once semantics, out-of-core state
> backends, and queryable state with CDC can have very strong use cases. For
> this feature there should only be an extra dependency to the AWS Java SDK for
> DynamoDB, which has Apache License 2.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10690) Tests leak resources via Files.list
[
https://issues.apache.org/jira/browse/FLINK-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669882#comment-16669882
]
ASF GitHub Bot commented on FLINK-10690:
zentol closed pull request #6939: [FLINK-10690][tests] Fix Files.list resource
leaks
URL: https://github.com/apache/flink/pull/6939
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/flink-core/src/test/java/org/apache/flink/util/FileUtilsTest.java
b/flink-core/src/test/java/org/apache/flink/util/FileUtilsTest.java
index 23878cbccb8..76b78056440 100644
--- a/flink-core/src/test/java/org/apache/flink/util/FileUtilsTest.java
+++ b/flink-core/src/test/java/org/apache/flink/util/FileUtilsTest.java
@@ -38,6 +38,7 @@
import java.util.Comparator;
import java.util.List;
import java.util.stream.Collectors;
+import java.util.stream.Stream;
import static org.junit.Assert.assertArrayEquals;
import static org.junit.Assert.assertEquals;
@@ -227,12 +228,18 @@ private static void assertDirEquals(java.nio.file.Path
expected, java.nio.file.P
assertEquals(expected.getFileName(), actual.getFileName());
if (Files.isDirectory(expected)) {
- List expectedContents =
Files.list(expected)
-
.sorted(Comparator.comparing(java.nio.file.Path::toString))
- .collect(Collectors.toList());
- List actualContents =
Files.list(actual)
-
.sorted(Comparator.comparing(java.nio.file.Path::toString))
- .collect(Collectors.toList());
+ List expectedContents;
+ try (Stream files =
Files.list(expected)) {
+ expectedContents = files
+
.sorted(Comparator.comparing(java.nio.file.Path::toString))
+ .collect(Collectors.toList());
+ }
+ List actualContents;
+ try (Stream files =
Files.list(actual)) {
+ actualContents = files
+
.sorted(Comparator.comparing(java.nio.file.Path::toString))
+ .collect(Collectors.toList());
+ }
assertEquals(expectedContents.size(),
actualContents.size());
diff --git
a/flink-end-to-end-tests/flink-distributed-cache-via-blob-test/src/main/java/org/apache/flink/streaming/tests/DistributedCacheViaBlobTestProgram.java
b/flink-end-to-end-tests/flink-distributed-cache-via-blob-test/src/main/java/org/apache/flink/streaming/tests/DistributedCacheViaBlobTestProgram.java
index 167101cc9aa..388cdc6adfa 100644
---
a/flink-end-to-end-tests/flink-distributed-cache-via-blob-test/src/main/java/org/apache/flink/streaming/tests/DistributedCacheViaBlobTestProgram.java
+++
b/flink-end-to-end-tests/flink-distributed-cache-via-blob-test/src/main/java/org/apache/flink/streaming/tests/DistributedCacheViaBlobTestProgram.java
@@ -26,6 +26,7 @@
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.stream.Collectors;
+import java.util.stream.Stream;
/**
* End-to-end test program for verifying that files are distributed via
BlobServer and later accessible through
@@ -47,8 +48,10 @@ public static void main(String[] args) throws Exception {
env.registerCachedFile(inputFile.toString(), "test_data",
false);
env.registerCachedFile(inputDir.toString(), "test_dir", false);
- Path containedFile = Files.list(inputDir).findAny()
- .orElseThrow(() -> new RuntimeException("Input
directory must not be empty."));
+ final Path containedFile;
+ try (Stream files = Files.list(inputDir)) {
+ containedFile = files.findAny().orElseThrow(() -> new
RuntimeException("Input directory must not be empty."));
+ }
env.fromElements(1)
.map(new TestMapFunction(
@@ -96,8 +99,10 @@ public String map(Integer value) throws Exception {
"initial dir. Input dir path: %s. Cache
dir path: %s", initialDirPath, testDir));
}
- if
(Files.list(testDir).map(Path::getFileName).map(Path::toString).noneMatch(path
-> path.equals(containedFileName))) {
- throw new
RuntimeException(String.format("Cached directory %s should not be empty.",
testDir));
+ try (Stream files = Files.list(testDir))
[GitHub] NicoK opened a new pull request #6971: [FLINK-9552][iterations] fix not syncing on checkpoint lock before emitting records
NicoK opened a new pull request #6971: [FLINK-9552][iterations] fix not syncing on checkpoint lock before emitting records URL: https://github.com/apache/flink/pull/6971 ## What is the purpose of the change We need to make sure that concurrent access to the `RecordWriter` is protected by a lock or otherwise if multiple threads access a `RecordWriter` instance at the same time, we may get into any type of data corruption which may not be recognised immediately. It seems that everything but the `StreamIterationHead` was synchronizing on the checkpoint lock and hence we should sync here as well. ## Brief change log - adapt `StreamIterationHead` to synchronize on the checkpoint lock when emitting records, watermarks, etc. ## Verifying this change This change is already covered by existing tests, and was tested manually by adding a check that `RecordWriter` interactions hold the checkpoint lock (by default, the `RecordWriter` does not know about this lock). ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** - The serializers: **no** - The runtime per-record code paths (performance sensitive): **yes** - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: **no** - The S3 file system connector: **no** ## Documentation - Does this pull request introduce a new feature? **no** - If yes, how is the feature documented? **not applicable** This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] StefanRRichter opened a new pull request #6972: [FLINK-9635][scheduling] Avoid task spread-out in scheduling with loc…
StefanRRichter opened a new pull request #6972: [FLINK-9635][scheduling] Avoid task spread-out in scheduling with loc… URL: https://github.com/apache/flink/pull/6972 …al recovery This PR is just a backport of FLINK-9635, PR #6961 to the release-1.6 branch. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-9635) Local recovery scheduling can cause spread out of tasks
[
https://issues.apache.org/jira/browse/FLINK-9635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669884#comment-16669884
]
ASF GitHub Bot commented on FLINK-9635:
---
StefanRRichter opened a new pull request #6972: [FLINK-9635][scheduling] Avoid
task spread-out in scheduling with loc…
URL: https://github.com/apache/flink/pull/6972
…al recovery
This PR is just a backport of FLINK-9635, PR #6961 to the release-1.6 branch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Local recovery scheduling can cause spread out of tasks
> ---
>
> Key: FLINK-9635
> URL: https://issues.apache.org/jira/browse/FLINK-9635
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination
>Affects Versions: 1.5.0, 1.6.2
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> In order to make local recovery work, Flink's scheduling was changed such
> that it tries to be rescheduled to its previous location. In order to not
> occupy slots which have state of other tasks cached, the strategy will
> request a new slot if the old slot identified by the previous allocation id
> is no longer present. This also applies to newly allocated slots because
> there is no distinction between new or already used. This behaviour can cause
> that every tasks gets deployed to its own slot if the {{SlotPool}} has
> released all slots in the meantime, for example. The consequence could be
> that a job can no longer be executed after a failure because it needs more
> slots than before.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9552) NPE in SpanningRecordSerializer during checkpoint with iterations
[
https://issues.apache.org/jira/browse/FLINK-9552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-9552:
--
Labels: pull-request-available (was: )
> NPE in SpanningRecordSerializer during checkpoint with iterations
> -
>
> Key: FLINK-9552
> URL: https://issues.apache.org/jira/browse/FLINK-9552
> Project: Flink
> Issue Type: Bug
> Components: Type Serialization System
>Affects Versions: 1.5.0
>Reporter: Truong Duc Kien
>Assignee: Nico Kruber
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> We're encountering NPE intermittently inside SpanningRecordSerializer during
> checkpoint.
>
> {code:java}
> 2018-06-08 08:31:35,741 [ka.actor.default-dispatcher-83] INFO
> o.a.f.r.e.ExecutionGraph IterationSource-22 (44/120)
> (c1b94ef849db0e5fb9fb7b85c17073ce) switched from RUNNING to FAILED.
> java.lang.RuntimeException
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89)
> at
> org.apache.flink.streaming.runtime.tasks.StreamIterationHead.run(StreamIterationHead.java:91)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:306)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:98)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:129)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:105)
> at
> org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:81)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:107)
> ... 5 more
> 2018-06-08 08:31:35,746 [ka.actor.default-dispatcher-83] INFO
> o.a.f.r.e.ExecutionGraph Job xxx (8a4eaf02c46dc21c7d6f3f70657dbb17) switched
> from state RUNNING to FAILING.
> java.lang.RuntimeException
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89)
> at
> org.apache.flink.streaming.runtime.tasks.StreamIterationHead.run(StreamIterationHead.java:91)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:306)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:98)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.sendToTarget(RecordWriter.java:129)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:105)
> at
> org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:81)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:107)
> ... 5 more
> {code}
> This issue is probably concurrency related, because the revelant Flink code
> seems to have proper null checking
> https://github.com/apache/flink/blob/fa024726bb801fc71cec5cc303cac1d4a03f555e/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/serialization/SpanningRecordSerializer.java#L98
> {code:java}
> // Copy from intermediate buffers to current target memory segment
> if (targetBuffer != null) {
> targetBuffer.append(lengthBuffer);
> targetBuffer.append(dataBuffer);
> targetBuffer.commit();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] sepulrator opened a new pull request #6973: [hotfix] [docs] fix typo in Java, Elasticsearch6 example
sepulrator opened a new pull request #6973: [hotfix] [docs] fix typo in Java, Elasticsearch6 example URL: https://github.com/apache/flink/pull/6973 Java, Elasticsearch 6.x ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
