[GitHub] flink pull request #:
Github user heytitle commented on the pull request: https://github.com/apache/flink/commit/ff3a35e93e858e30debcb1e85cc056045c1fe2a0#commitcomment-24687576 ð ---
[GitHub] flink pull request #:
Github user heytitle commented on the pull request: https://github.com/apache/flink/commit/9016cce503b4d471b5a49f0abccc196945ada97e#commitcomment-24529391 In flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java: In flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java on line 188: Can we simplify it to the code below? ``` Class generatedClass = null WeakReference fromCache = generatedClassCache.getOrDefault(cacheKey, null); generatedClass = fromCache != null ? fromCache.get() : null; if ( genenetedClass == null ) { // cache miss ... } ``` So, we don't need to introduce `cacheHit` variable. ---
[GitHub] flink pull request #:
Github user heytitle commented on the pull request: https://github.com/apache/flink/commit/cce40c5cd4aaff446bb4bec8918d2fda37649e0a#commitcomment-24528598 In flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java: In flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java on line 180: Is it possible that the class will be disappear after the first `get()` from if's condition? ---
[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...
Github user heytitle commented on a diff in the pull request: https://github.com/apache/flink/pull/3511#discussion_r140664033 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java --- @@ -0,0 +1,198 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.runtime.codegeneration; + +import org.apache.flink.api.common.ExecutionConfig; +import org.apache.flink.api.common.typeutils.TypeComparator; +import org.apache.flink.api.common.typeutils.TypeSerializer; +import org.apache.flink.core.memory.MemorySegment; +import org.apache.flink.runtime.operators.sort.FixedLengthRecordSorter; +import org.apache.flink.runtime.operators.sort.InMemorySorter; +import org.apache.flink.runtime.operators.sort.NormalizedKeySorter; + +import freemarker.template.Configuration; +import freemarker.template.Template; +import freemarker.template.TemplateException; +import freemarker.template.TemplateExceptionHandler; +import freemarker.template.Version; +import org.codehaus.commons.compiler.CompileException; +import org.codehaus.janino.SimpleCompiler; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.io.StringWriter; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.util.HashMap; +import java.util.List; + +/** + * {@link SorterFactory} is a singleton class that provides functionalities to create the most suitable sorter + * for underlying data based on {@link TypeComparator}. + * Note: the generated code can be inspected by configuring Janino to write the code that is being compiled + * to a file, see http://janino-compiler.github.io/janino/#debugging + */ +public class SorterFactory { + // + // Constants + // + private static final Logger LOG = LoggerFactory.getLogger(SorterFactory.class); + + /** Fixed length records with a length below this threshold will be in-place sorted, if possible. */ + private static final int THRESHOLD_FOR_IN_PLACE_SORTING = 32; + + // + // Singleton Attribute + // + private static SorterFactory sorterFactory; + + // + // Attributes + // + private SimpleCompiler classCompiler; + private HashMap<String, Constructor> constructorCache; + private final Template template; + + /** +* This is only for testing. If an error occurs, we want to fail the test, instead of falling back +* to a non-generated sorter. +*/ + public boolean forceCodeGeneration = false; + + /** +* Constructor. +*/ + private SorterFactory() { + this.classCompiler = new SimpleCompiler(); + this.classCompiler.setParentClassLoader(this.getClass().getClassLoader()); --- End diff -- Ok, I got it now. I thought there is some things behind this parent class loader field and influence the existence of `classComplier`. ---
[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...
Github user heytitle commented on a diff in the pull request: https://github.com/apache/flink/pull/3511#discussion_r140662538 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java --- @@ -0,0 +1,198 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.runtime.codegeneration; + +import org.apache.flink.api.common.ExecutionConfig; +import org.apache.flink.api.common.typeutils.TypeComparator; +import org.apache.flink.api.common.typeutils.TypeSerializer; +import org.apache.flink.core.memory.MemorySegment; +import org.apache.flink.runtime.operators.sort.FixedLengthRecordSorter; +import org.apache.flink.runtime.operators.sort.InMemorySorter; +import org.apache.flink.runtime.operators.sort.NormalizedKeySorter; + +import freemarker.template.Configuration; +import freemarker.template.Template; +import freemarker.template.TemplateException; +import freemarker.template.TemplateExceptionHandler; +import freemarker.template.Version; +import org.codehaus.commons.compiler.CompileException; +import org.codehaus.janino.SimpleCompiler; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.io.StringWriter; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.util.HashMap; +import java.util.List; + +/** + * {@link SorterFactory} is a singleton class that provides functionalities to create the most suitable sorter + * for underlying data based on {@link TypeComparator}. + * Note: the generated code can be inspected by configuring Janino to write the code that is being compiled + * to a file, see http://janino-compiler.github.io/janino/#debugging + */ +public class SorterFactory { + // + // Constants + // + private static final Logger LOG = LoggerFactory.getLogger(SorterFactory.class); + + /** Fixed length records with a length below this threshold will be in-place sorted, if possible. */ + private static final int THRESHOLD_FOR_IN_PLACE_SORTING = 32; + + // + // Singleton Attribute + // + private static SorterFactory sorterFactory; + + // + // Attributes + // + private SimpleCompiler classCompiler; + private HashMap<String, Constructor> constructorCache; + private final Template template; + + /** +* This is only for testing. If an error occurs, we want to fail the test, instead of falling back +* to a non-generated sorter. +*/ + public boolean forceCodeGeneration = false; + + /** +* Constructor. +*/ + private SorterFactory() { + this.classCompiler = new SimpleCompiler(); + this.classCompiler.setParentClassLoader(this.getClass().getClassLoader()); --- End diff -- Thanks for the comment. I have a further comment on this What I understand is that If `SorterFactory` is a singleton object and its parent class loader isn't the user one, it might happen that the next job will fail to create new code-generated sorters. This is due to the fact that `classComplier` is removed after the first job finishes and it is instantiated only once when `SorterFactory is created. Please correct me if I'm wrong. ---
[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...
Github user heytitle commented on a diff in the pull request: https://github.com/apache/flink/pull/3511#discussion_r140659544 --- Diff: flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/DriverBaseITCase.java --- @@ -190,6 +190,7 @@ private String getSystemOutput(String[] args) throws Exception { switch (mode) { case CLUSTER: case COLLECTION: + case CLUSTER_WITH_CODEGENERATION_ENABLED: --- End diff -- First of all, I'm not sure whether this is a good way to get this estimation. I estimated the build time by running all tests from `flink-gelly-examples` inside IntelliJ IDEA. With `CLUSTER_WITH_CODEGENERATION_ENABLED` : `2m 20s` Without `CLUSTER_WITH_CODEGENERATION_ENABLED`: `1m 27s` Patch for disabling `CLUSTER_WITH_CODEGENERATION_ENABLED` case : https://gist.github.com/heytitle/89961fcaabcf326eadee190b9d6085a6 ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 @ggevay Thanks for the commits. They look good. I also added you to my repository. For `FixedLengthRecordSorter`, I agree with you that we should implement that when #2617 is merged. How does feature freeze work? Apart from getting this PR merged, is there anything else we have to do? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 Hi @ggevay, Thanks for the commits. Do you have any plan for `FixedLengthRecordSorter` implementation? I'm not sure how much work need to be done there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 Hi @ggevay, I have rebased master and resolved all coding style related issues and pushed the code to https://github.com/heytitle/flink/tree/FLINK-5734-rebase-master-temp For the next step, I'm trying to pass `mvn clean verify`. Right now, some tests are randomly failed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 Hi @ggevay, I'm currently fixing coding style. Maybe you can help me take a look at what Greg suggested related `FixedLengthRecordSorter` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 Hi @greghogan, Thank very much for the feedback. > Are you able to continue working on this feature? Yes, I would like to complete the feature and will take a look into the issues you mentioned in next couple of weeks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...
Github user heytitle commented on a diff in the pull request: https://github.com/apache/flink/pull/3511#discussion_r115130267 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/NormalizedKeySorter.java --- @@ -47,61 +47,61 @@ public static final int MAX_NORMALIZED_KEY_LEN_PER_ELEMENT = 8; - private static final int MIN_REQUIRED_BUFFERS = 3; + public static final int MIN_REQUIRED_BUFFERS = 3; - private static final int LARGE_RECORD_THRESHOLD = 10 * 1024 * 1024; + public static final int LARGE_RECORD_THRESHOLD = 10 * 1024 * 1024; - private static final long LARGE_RECORD_TAG = 1L << 63; + public static final long LARGE_RECORD_TAG = 1L << 63; - private static final long POINTER_MASK = LARGE_RECORD_TAG - 1; + public static final long POINTER_MASK = LARGE_RECORD_TAG - 1; --- End diff -- The reason `public` is used here because `Janino` first check accessibility of these variables and it seems not able to access them when `protected` is used and it throws the error below. ``` org.codehaus.commons.compiler.CompileException: Field "LARGE_RECORD_THRESHOLD" is not accessible at org.codehaus.janino.ReflectionIClass$ReflectionIField.getConstantValue(ReflectionIClass.java:340) at org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:4433) at org.codehaus.janino.UnitCompiler.access$1(UnitCompiler.java:182) at org.codehaus.janino.UnitCompiler$11.visitFieldAccess(UnitCompiler.java:4407) at org.codehaus.janino.Java$FieldAccess.accept(Java.java:3229) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 Hi @greghogan, Thanks for the explanation. I like the idea. I also think we might not need `NormalizedKeySorterBase`, we can just extend generated sorters from `NormalizedKeySorter` and override those methods . What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3779: [FLINK-6382] [gelly] Support all numeric types for...
Github user heytitle commented on a diff in the pull request: https://github.com/apache/flink/pull/3779#discussion_r114009718 --- Diff: flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/DriverBaseITCase.java --- @@ -41,17 +43,41 @@ @Rule public ExpectedException expectedException = ExpectedException.none(); - protected DriverBaseITCase(TestExecutionMode mode) { + protected final String idType; + + protected DriverBaseITCase(TestExecutionMode mode, String idType) { super(mode); + + this.idType = idType; } - // extend MultipleProgramsTestBase default to include object reuse mode - @Parameterized.Parameters(name = "Execution mode = {0}") + @Parameterized.Parameters(name = "Execution mode = {0}, ID type = {1}") public static Collection<Object[]> executionModes() { - return Arrays.asList( - new Object[] { TestExecutionMode.CLUSTER }, - new Object[] { TestExecutionMode.CLUSTER_OBJECT_REUSE }, - new Object[] { TestExecutionMode.COLLECTION }); + List<Object[]> executionModes = new ArrayList<>(); + + for (TestExecutionMode executionMode : TestExecutionMode.values()) { + for (String idType : new String[] {"byte", "nativeByte", "short", "nativeShort", "char", "nativeChar", + "integer", "nativeInteger", "long", "nativeLong", "string", "nativeString"}) { --- End diff -- What do you think if we declare the list of `idType` before the outer loop? IMHO, this will make formation better. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3779: [FLINK-6382] [gelly] Support all numeric types for...
Github user heytitle commented on a diff in the pull request: https://github.com/apache/flink/pull/3779#discussion_r114010412 --- Diff: flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/library/link_analysis/HITS.java --- @@ -60,8 +60,6 @@ * * http://www.cs.cornell.edu/home/kleinber/auth.pdf * - * http://www.cs.cornell.edu/home/kleinber/auth.pdf - * --- End diff -- Nice catch! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3779: [FLINK-6382] [gelly] Support all numeric types for...
Github user heytitle commented on a diff in the pull request: https://github.com/apache/flink/pull/3779#discussion_r114007404 --- Diff: flink-libraries/flink-gelly-examples/src/main/java/org/apache/flink/graph/drivers/input/GridGraph.java --- @@ -82,11 +81,27 @@ public void configure(ParameterTool parameterTool) throws ProgramParametrization @Override public String getIdentity() { - return getName() + " (" + dimensions + ")"; + return getTypeName() + " " + getName() + " (" + dimensions + ")"; } @Override - public Graph<LongValue, NullValue, NullValue> create(ExecutionEnvironment env) { + protected long vertexCount() { + // in Java 8 use Math.multiplyExact(long, long) --- End diff -- What is the purpose of this comment? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 I also think about the abstract class but I'm not sure how to do it properly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 @greghogan May I ask you how to remove `FLINK-3722` commits?. Only way I can think of is `git rebase -i`, but this will rewrite history of this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 I haven't posted it yet. I will do it next week. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3511: Flink 5734 code generation for normalizedkey sorte...
GitHub user heytitle opened a pull request: https://github.com/apache/flink/pull/3511 Flink 5734 code generation for normalizedkey sorter You can merge this pull request into a Git repository by running: $ git pull https://github.com/heytitle/flink FLINK-5734-code-generation-for-normalizedkey-sorter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3511.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3511 commit dd20fb63d0ef64de11ba5435cc9bcd3fab106dde Author: Greg Hogan <c...@greghogan.com> Date: 2016-10-05T20:13:02Z [FLINK-3722] [runtime] Don't / and % when sorting Replace division and modulus with addition and subtraction. commit 4d87bbb8d34f9551d10cb0213fda842761f11be6 Author: Greg Hogan <c...@greghogan.com> Date: 2016-10-20T13:40:27Z Additional comments commit fc597eb6949c502ae7883bd962feaa6b129b8172 Author: heytitle <pat.chor...@gmail.com> Date: 2017-02-19T20:28:29Z [FLINK-5734] prepare project structure commit ecfcd730049317d0521b25c9474c286d97322db8 Author: heytitle <pat.chor...@gmail.com> Date: 2017-02-20T14:22:41Z [FLINK-5734] implement basic functionalities for code generation commit 46bcebd0fc188c1a2d2e714072a75f76db2d8232 Author: heytitle <pat.chor...@gmail.com> Date: 2017-02-24T22:17:50Z [FLINK-5734] use NormaliKeysorter's appoach to compute numKeyByte commit 8f3e0534c3dd1677c34b975b1126257bbb685f31 Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-04T21:55:21Z [FLINK-5734] use synchronized block for SorterFactory and TemplateModel commit 64ad5df842ca14e929b893c76a790827eb1d0b90 Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-05T00:01:54Z [FLINK-5734] replace string concat with stringbuilder commit d81ed5076fad7112d53d40ecf148871254fa57f3 Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-05T00:34:08Z [FLINK-5734] add testcase for variable-length string commit 237fee4de158021d51bbf2c0550933f41cd90aab Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-05T00:48:47Z [FLINK-5734] user proper logger and also add comments commit 3b38a6f9dee406436864af7c6052db84a9ebfd34 Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-05T16:29:16Z [FLINK-5734] checking endianness should not be involved in generating procedures for compare function commit 559daf5d48f1d786de72127c92cb1c1f6a4c2ff9 Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-05T20:12:50Z [FLINK-5734] move byte-operator-mapping to be a constant of class scope commit 56454f63ebf23ea962c793ac0dc35493bc5629cc Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-05T21:04:17Z [FLINK-5734] add enable/disable flag in ExecutionConfig commit 912641c4e3894b9c4cbfe82c2ac86fc96d463725 Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-05T22:09:20Z [FLINK-5734] rename variable names in old compare/swap functions to match with the new ones commit bc8bb29d3aec66908aa3e1e91fd5ea2661afd1f8 Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-05T22:09:49Z [FLINK-5734] improve tests commit 45d0aa590b21d0c23d76601f5e8efe6f17c0211b Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-06T15:14:54Z [FLINK-5734] add timestamp to generated sorter commit 3563e213bdb0d54bcee3c64e753f4656862554f3 Author: SerkanAli <erg...@campus.tu-berlin.de> Date: 2017-03-07T13:27:44Z [FLINK-5734] RESOURCE_PATH exhibits to a Temporary directory commit 8d3e08485931c0d0d5bcf6647eb346eff65c6fa7 Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-07T22:35:25Z [FLINK-5734] integrate code generation to flink code commit 362fc4656f60354e4ec23732ec339067adb47b1e Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-08T00:26:34Z [FLINK-5734] prevent generating same sorter at the same time commit 3415cbaf0dc8d9ecaf0f1c9652b15a49eececca2 Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-08T00:35:38Z [FLINK-5734] fix sorting in desc order failed commit bcf50d285e2040443e1b9857484b63008304c9c5 Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-08T12:03:19Z [FLINK-5734] also cache constructor to save cooking time commit bc2a37c4b6e41428f320efc9fd70e3b47f6aad5e Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-08T14:34:25Z [FLINK-5734] refactor integration tests commit f06f46873587b21357a86897ca5096a0dcb06a20 Author: heytitle <pat.chor...@gmail.com> Date: 2017-03-08T23:09:08Z [FLINK-5734] add benchmarking code commit 7c8f82f3c11bfba3a20b34fe5315a9d0cb827805 Author: SerkanAli <erg...@campus.tu-berlin.de> Date: 2017-03-09T12:24:52Z [FLINK-5734] get temporary directory for generated code from task config. --- If your proj
[GitHub] flink pull request #3393: [FLINK-3903][docs] adding alternative installation...
Github user heytitle commented on a diff in the pull request: https://github.com/apache/flink/pull/3393#discussion_r102675770 --- Diff: docs/quickstart/setup_quickstart.md --- @@ -72,6 +72,15 @@ $ cd build-target # this is where Flink is installed to ~~~ {% endif %} +### Alternatively --- End diff -- Should we also put to OSX installation into the if/else? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3393: [FLINK-3903][docs] adding alternative installation...
GitHub user heytitle opened a pull request: https://github.com/apache/flink/pull/3393 [FLINK-3903][docs] adding alternative installation documentation for ⦠â¦OSX users Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html). In addition to going through the list, please provide a meaningful description of your changes. - [x] General - The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text") - The pull request addresses only one issue - Each commit in the PR has a meaningful commit message (including the JIRA id) - [x] Documentation - Documentation has been added for new functionality - Old documentation affected by the pull request has been updated - JavaDoc for public methods has been added - [x] Tests & Build - Functionality added by the pull request is covered by tests - `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/heytitle/flink FLINK-3903-homebrew-installation Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3393.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3393 commit 0539e3f72e19ba5523410b4b03bd446969c8fbb7 Author: heytitle <pat.chor...@gmail.com> Date: 2017-02-22T16:04:42Z [FLINK-3903][docs] adding alternative installation documentation for OSX users --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3080: [FLINK-4920] Add a Scala Function Gauge
Github user heytitle commented on a diff in the pull request: https://github.com/apache/flink/pull/3080#discussion_r95671339 --- Diff: flink-scala/src/main/scala/org/apache/flink/api/scala/metrics/ScalaGauge.scala --- @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.api.scala.metrics + +import org.apache.flink.metrics.Gauge + --- End diff -- IMHO, this class is quite small and obvious what it is doing. However, if you think it it better to have some explanation, here is my idea. ``` A ScalaGauge is similar to a Gauge in {@link Metric} but made for Scala API. ``` What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3080: [FLINK-4920] Add a Scala Function Gauge
Github user heytitle commented on a diff in the pull request: https://github.com/apache/flink/pull/3080#discussion_r95666996 --- Diff: flink-scala/src/test/scala/org/apache/flink/api/scala/metrics/ScalaGaugeTest.scala --- @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.api.scala.metrics + +import org.apache.flink.metrics.Gauge +import org.apache.flink.runtime.metrics.{MetricRegistry, MetricRegistryConfiguration} +import org.apache.flink.runtime.metrics.groups.GenericMetricGroup +import org.apache.flink.util.TestLogger +import org.junit.Test +import org.scalatest.junit.JUnitSuiteLike + +class ScalaGaugeTest extends TestLogger with JUnitSuiteLike { + + @Test + def testGaugeCorrectValue(): Unit = { +val myGauge = ScalaGauge[Long]( () => 4 ) +assert( myGauge.getValue == 4 ) + } + + @Test + def testRegister(): Unit = { +val conf = MetricRegistryConfiguration.defaultMetricRegistryConfiguration() +val registry = new MetricRegistry(conf) +val metricGroup = new GenericMetricGroup(registry, null, "world") + +val myGauge = ScalaGauge[Long]( () => 4 ) +val gauge = metricGroup.gauge[Long, Gauge[Long]]("max", myGauge); + +assert( gauge == myGauge ) --- End diff -- yeah, that's true. I'll remove it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3080: [FLINK-4920] Add a Scala Function Gauge
Github user heytitle commented on a diff in the pull request: https://github.com/apache/flink/pull/3080#discussion_r95347426 --- Diff: flink-scala/src/main/scala/org/apache/flink/api/scala/metrics/ScalaGauge.scala --- @@ -0,0 +1,27 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.api.scala.metrics + +import org.apache.flink.metrics.Gauge + +class ScalaGauge[T](value : T) extends Gauge[T] { --- End diff -- @KurtYoung Thank for the comment. I see your point. I'll fix it accordingly. Also, I have an idea of having a companion object of `ScalaGauge`. As a result, we can instantiate `ScalaGauge` without calling `new`. What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3080: [FLINK-4920] Add a Scala Function Gauge
GitHub user heytitle opened a pull request: https://github.com/apache/flink/pull/3080 [FLINK-4920] Add a Scala Function Gauge Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html). In addition to going through the list, please provide a meaningful description of your changes. - [x] General - The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text") - The pull request addresses only one issue - Each commit in the PR has a meaningful commit message (including the JIRA id) - [x] Documentation - Documentation has been added for new functionality - Old documentation affected by the pull request has been updated - JavaDoc for public methods has been added - [x] Tests & Build [![Build Status](https://travis-ci.org/heytitle/flink.svg?branch=scala-gauge)](https://travis-ci.org/heytitle/flink) - Functionality added by the pull request is covered by tests - `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/heytitle/flink scala-gauge Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3080.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3080 commit 1a464e2b74930c63aada3dbae40b74822ccea89c Author: heytitle <pat.chor...@gmail.com> Date: 2016-12-27T22:21:19Z [FLINK-4920] Introduce Scala Function Gauge commit aa9b7bb435e97687a8d69820579bddf41b1b29a4 Author: heytitle <pat.chor...@gmail.com> Date: 2017-01-07T10:30:40Z [FLINK-4920] Update gauge document --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2611: [FLINK-4778] Update program example in /docs/setup/cli.md...
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/2611 @fhueske you're welcome ð --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2611: Update the way we call WordCount program in setup/...
GitHub user heytitle opened a pull request: https://github.com/apache/flink/pull/2611 Update the way we call WordCount program in setup/cli.md examples. Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html). In addition to going through the list, please provide a meaningful description of your changes. - [ ] General - The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text") - The pull request addresses only one issue - Each commit in the PR has a meaningful commit message (including the JIRA id) - [ ] Documentation - Documentation has been added for new functionality - Old documentation affected by the pull request has been updated - JavaDoc for public methods has been added - [ ] Tests & Build - Functionality added by the pull request is covered by tests - `mvn clean verify` has been executed successfully locally or a Travis build has passed Due to the change in https://github.com/apache/flink/commit/0629e25602eefdc239e8e72d9e3c9c1a5164448e, we need to specify a prefix for `input` and `output` files. You can merge this pull request into a Git repository by running: $ git pull https://github.com/heytitle/flink update-doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2611.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2611 commit f15d38850d1980d7a3e78edf54189483ee880a22 Author: Pattarawat Chormai <pat.chor...@gmail.com> Date: 2016-10-07T19:34:33Z Update the way we call WordCount program. Due to the change in https://github.com/apache/flink/commit/0629e25602eefdc239e8e72d9e3c9c1a5164448e, we need to specify a prefix for `input` and `output` files. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---