[GitHub] flink pull request #:
Github user heytitle commented on the pull request: https://github.com/apache/flink/commit/ff3a35e93e858e30debcb1e85cc056045c1fe2a0#commitcomment-24687576 ð ---
[GitHub] flink pull request #:
Github user heytitle commented on the pull request:
https://github.com/apache/flink/commit/9016cce503b4d471b5a49f0abccc196945ada97e#commitcomment-24529391
In
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java:
In
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java
on line 188:
Can we simplify it to the code below?
```
Class generatedClass = null
WeakReference fromCache = generatedClassCache.getOrDefault(cacheKey,
null);
generatedClass = fromCache != null ? fromCache.get() : null;
if ( genenetedClass == null ) {
// cache miss
...
}
```
So, we don't need to introduce `cacheHit` variable.
---
[GitHub] flink pull request #:
Github user heytitle commented on the pull request: https://github.com/apache/flink/commit/cce40c5cd4aaff446bb4bec8918d2fda37649e0a#commitcomment-24528598 In flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java: In flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java on line 180: Is it possible that the class will be disappear after the first `get()` from if's condition? ---
[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...
Github user heytitle commented on a diff in the pull request:
https://github.com/apache/flink/pull/3511#discussion_r140664033
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java
---
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.codegeneration;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.operators.sort.FixedLengthRecordSorter;
+import org.apache.flink.runtime.operators.sort.InMemorySorter;
+import org.apache.flink.runtime.operators.sort.NormalizedKeySorter;
+
+import freemarker.template.Configuration;
+import freemarker.template.Template;
+import freemarker.template.TemplateException;
+import freemarker.template.TemplateExceptionHandler;
+import freemarker.template.Version;
+import org.codehaus.commons.compiler.CompileException;
+import org.codehaus.janino.SimpleCompiler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashMap;
+import java.util.List;
+
+/**
+ * {@link SorterFactory} is a singleton class that provides
functionalities to create the most suitable sorter
+ * for underlying data based on {@link TypeComparator}.
+ * Note: the generated code can be inspected by configuring Janino to
write the code that is being compiled
+ * to a file, see http://janino-compiler.github.io/janino/#debugging
+ */
+public class SorterFactory {
+ //
+ // Constants
+ //
+ private static final Logger LOG =
LoggerFactory.getLogger(SorterFactory.class);
+
+ /** Fixed length records with a length below this threshold will be
in-place sorted, if possible. */
+ private static final int THRESHOLD_FOR_IN_PLACE_SORTING = 32;
+
+ //
+ // Singleton Attribute
+ //
+ private static SorterFactory sorterFactory;
+
+ //
+ // Attributes
+ //
+ private SimpleCompiler classCompiler;
+ private HashMap<String, Constructor> constructorCache;
+ private final Template template;
+
+ /**
+* This is only for testing. If an error occurs, we want to fail the
test, instead of falling back
+* to a non-generated sorter.
+*/
+ public boolean forceCodeGeneration = false;
+
+ /**
+* Constructor.
+*/
+ private SorterFactory() {
+ this.classCompiler = new SimpleCompiler();
+
this.classCompiler.setParentClassLoader(this.getClass().getClassLoader());
--- End diff --
Ok, I got it now. I thought there is some things behind this parent class
loader field and influence the existence of `classComplier`.
---
[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...
Github user heytitle commented on a diff in the pull request:
https://github.com/apache/flink/pull/3511#discussion_r140662538
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/codegeneration/SorterFactory.java
---
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.codegeneration;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.operators.sort.FixedLengthRecordSorter;
+import org.apache.flink.runtime.operators.sort.InMemorySorter;
+import org.apache.flink.runtime.operators.sort.NormalizedKeySorter;
+
+import freemarker.template.Configuration;
+import freemarker.template.Template;
+import freemarker.template.TemplateException;
+import freemarker.template.TemplateExceptionHandler;
+import freemarker.template.Version;
+import org.codehaus.commons.compiler.CompileException;
+import org.codehaus.janino.SimpleCompiler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashMap;
+import java.util.List;
+
+/**
+ * {@link SorterFactory} is a singleton class that provides
functionalities to create the most suitable sorter
+ * for underlying data based on {@link TypeComparator}.
+ * Note: the generated code can be inspected by configuring Janino to
write the code that is being compiled
+ * to a file, see http://janino-compiler.github.io/janino/#debugging
+ */
+public class SorterFactory {
+ //
+ // Constants
+ //
+ private static final Logger LOG =
LoggerFactory.getLogger(SorterFactory.class);
+
+ /** Fixed length records with a length below this threshold will be
in-place sorted, if possible. */
+ private static final int THRESHOLD_FOR_IN_PLACE_SORTING = 32;
+
+ //
+ // Singleton Attribute
+ //
+ private static SorterFactory sorterFactory;
+
+ //
+ // Attributes
+ //
+ private SimpleCompiler classCompiler;
+ private HashMap<String, Constructor> constructorCache;
+ private final Template template;
+
+ /**
+* This is only for testing. If an error occurs, we want to fail the
test, instead of falling back
+* to a non-generated sorter.
+*/
+ public boolean forceCodeGeneration = false;
+
+ /**
+* Constructor.
+*/
+ private SorterFactory() {
+ this.classCompiler = new SimpleCompiler();
+
this.classCompiler.setParentClassLoader(this.getClass().getClassLoader());
--- End diff --
Thanks for the comment. I have a further comment on this
What I understand is that If `SorterFactory` is a singleton object and its
parent class loader isn't the user one, it might happen that the next job will
fail to create new code-generated sorters. This is due to the fact that
`classComplier` is removed after the first job finishes and it is instantiated
only once when `SorterFactory is created.
Please correct me if I'm wrong.
---
[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...
Github user heytitle commented on a diff in the pull request:
https://github.com/apache/flink/pull/3511#discussion_r140659544
--- Diff:
flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/DriverBaseITCase.java
---
@@ -190,6 +190,7 @@ private String getSystemOutput(String[] args) throws
Exception {
switch (mode) {
case CLUSTER:
case COLLECTION:
+ case CLUSTER_WITH_CODEGENERATION_ENABLED:
--- End diff --
First of all, I'm not sure whether this is a good way to get this
estimation.
I estimated the build time by running all tests from `flink-gelly-examples`
inside IntelliJ IDEA.
With `CLUSTER_WITH_CODEGENERATION_ENABLED` : `2m 20s`
Without `CLUSTER_WITH_CODEGENERATION_ENABLED`: `1m 27s`
Patch for disabling `CLUSTER_WITH_CODEGENERATION_ENABLED` case :
https://gist.github.com/heytitle/89961fcaabcf326eadee190b9d6085a6
---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 @ggevay Thanks for the commits. They look good. I also added you to my repository. For `FixedLengthRecordSorter`, I agree with you that we should implement that when #2617 is merged. How does feature freeze work? Apart from getting this PR merged, is there anything else we have to do? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 Hi @ggevay, Thanks for the commits. Do you have any plan for `FixedLengthRecordSorter` implementation? I'm not sure how much work need to be done there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 Hi @ggevay, I have rebased master and resolved all coding style related issues and pushed the code to https://github.com/heytitle/flink/tree/FLINK-5734-rebase-master-temp For the next step, I'm trying to pass `mvn clean verify`. Right now, some tests are randomly failed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 Hi @ggevay, I'm currently fixing coding style. Maybe you can help me take a look at what Greg suggested related `FixedLengthRecordSorter` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 Hi @greghogan, Thank very much for the feedback. > Are you able to continue working on this feature? Yes, I would like to complete the feature and will take a look into the issues you mentioned in next couple of weeks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3511: [Flink-5734] code generation for normalizedkey sor...
Github user heytitle commented on a diff in the pull request: https://github.com/apache/flink/pull/3511#discussion_r115130267 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/operators/sort/NormalizedKeySorter.java --- @@ -47,61 +47,61 @@ public static final int MAX_NORMALIZED_KEY_LEN_PER_ELEMENT = 8; - private static final int MIN_REQUIRED_BUFFERS = 3; + public static final int MIN_REQUIRED_BUFFERS = 3; - private static final int LARGE_RECORD_THRESHOLD = 10 * 1024 * 1024; + public static final int LARGE_RECORD_THRESHOLD = 10 * 1024 * 1024; - private static final long LARGE_RECORD_TAG = 1L << 63; + public static final long LARGE_RECORD_TAG = 1L << 63; - private static final long POINTER_MASK = LARGE_RECORD_TAG - 1; + public static final long POINTER_MASK = LARGE_RECORD_TAG - 1; --- End diff -- The reason `public` is used here because `Janino` first check accessibility of these variables and it seems not able to access them when `protected` is used and it throws the error below. ``` org.codehaus.commons.compiler.CompileException: Field "LARGE_RECORD_THRESHOLD" is not accessible at org.codehaus.janino.ReflectionIClass$ReflectionIField.getConstantValue(ReflectionIClass.java:340) at org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:4433) at org.codehaus.janino.UnitCompiler.access$1(UnitCompiler.java:182) at org.codehaus.janino.UnitCompiler$11.visitFieldAccess(UnitCompiler.java:4407) at org.codehaus.janino.Java$FieldAccess.accept(Java.java:3229) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 Hi @greghogan, Thanks for the explanation. I like the idea. I also think we might not need `NormalizedKeySorterBase`, we can just extend generated sorters from `NormalizedKeySorter` and override those methods . What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3779: [FLINK-6382] [gelly] Support all numeric types for...
Github user heytitle commented on a diff in the pull request:
https://github.com/apache/flink/pull/3779#discussion_r114009718
--- Diff:
flink-libraries/flink-gelly-examples/src/test/java/org/apache/flink/graph/drivers/DriverBaseITCase.java
---
@@ -41,17 +43,41 @@
@Rule
public ExpectedException expectedException = ExpectedException.none();
- protected DriverBaseITCase(TestExecutionMode mode) {
+ protected final String idType;
+
+ protected DriverBaseITCase(TestExecutionMode mode, String idType) {
super(mode);
+
+ this.idType = idType;
}
- // extend MultipleProgramsTestBase default to include object reuse mode
- @Parameterized.Parameters(name = "Execution mode = {0}")
+ @Parameterized.Parameters(name = "Execution mode = {0}, ID type = {1}")
public static Collection<Object[]> executionModes() {
- return Arrays.asList(
- new Object[] { TestExecutionMode.CLUSTER },
- new Object[] { TestExecutionMode.CLUSTER_OBJECT_REUSE },
- new Object[] { TestExecutionMode.COLLECTION });
+ List<Object[]> executionModes = new ArrayList<>();
+
+ for (TestExecutionMode executionMode :
TestExecutionMode.values()) {
+ for (String idType : new String[] {"byte",
"nativeByte", "short", "nativeShort", "char", "nativeChar",
+
"integer", "nativeInteger", "long", "nativeLong", "string", "nativeString"}) {
--- End diff --
What do you think if we declare the list of `idType` before the outer loop?
IMHO, this will make formation better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3779: [FLINK-6382] [gelly] Support all numeric types for...
Github user heytitle commented on a diff in the pull request: https://github.com/apache/flink/pull/3779#discussion_r114010412 --- Diff: flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/library/link_analysis/HITS.java --- @@ -60,8 +60,6 @@ * * http://www.cs.cornell.edu/home/kleinber/auth.pdf * - * http://www.cs.cornell.edu/home/kleinber/auth.pdf - * --- End diff -- Nice catch! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3779: [FLINK-6382] [gelly] Support all numeric types for...
Github user heytitle commented on a diff in the pull request:
https://github.com/apache/flink/pull/3779#discussion_r114007404
--- Diff:
flink-libraries/flink-gelly-examples/src/main/java/org/apache/flink/graph/drivers/input/GridGraph.java
---
@@ -82,11 +81,27 @@ public void configure(ParameterTool parameterTool)
throws ProgramParametrization
@Override
public String getIdentity() {
- return getName() + " (" + dimensions + ")";
+ return getTypeName() + " " + getName() + " (" + dimensions +
")";
}
@Override
- public Graph<LongValue, NullValue, NullValue>
create(ExecutionEnvironment env) {
+ protected long vertexCount() {
+ // in Java 8 use Math.multiplyExact(long, long)
--- End diff --
What is the purpose of this comment?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 I also think about the abstract class but I'm not sure how to do it properly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 @greghogan May I ask you how to remove `FLINK-3722` commits?. Only way I can think of is `git rebase -i`, but this will rewrite history of this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3511: [Flink-5734] code generation for normalizedkey sorter
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/3511 I haven't posted it yet. I will do it next week. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3511: Flink 5734 code generation for normalizedkey sorte...
GitHub user heytitle opened a pull request:
https://github.com/apache/flink/pull/3511
Flink 5734 code generation for normalizedkey sorter
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/heytitle/flink
FLINK-5734-code-generation-for-normalizedkey-sorter
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3511.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3511
commit dd20fb63d0ef64de11ba5435cc9bcd3fab106dde
Author: Greg Hogan <c...@greghogan.com>
Date: 2016-10-05T20:13:02Z
[FLINK-3722] [runtime] Don't / and % when sorting
Replace division and modulus with addition and subtraction.
commit 4d87bbb8d34f9551d10cb0213fda842761f11be6
Author: Greg Hogan <c...@greghogan.com>
Date: 2016-10-20T13:40:27Z
Additional comments
commit fc597eb6949c502ae7883bd962feaa6b129b8172
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-02-19T20:28:29Z
[FLINK-5734] prepare project structure
commit ecfcd730049317d0521b25c9474c286d97322db8
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-02-20T14:22:41Z
[FLINK-5734] implement basic functionalities for code generation
commit 46bcebd0fc188c1a2d2e714072a75f76db2d8232
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-02-24T22:17:50Z
[FLINK-5734] use NormaliKeysorter's appoach to compute numKeyByte
commit 8f3e0534c3dd1677c34b975b1126257bbb685f31
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-04T21:55:21Z
[FLINK-5734] use synchronized block for SorterFactory and TemplateModel
commit 64ad5df842ca14e929b893c76a790827eb1d0b90
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-05T00:01:54Z
[FLINK-5734] replace string concat with stringbuilder
commit d81ed5076fad7112d53d40ecf148871254fa57f3
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-05T00:34:08Z
[FLINK-5734] add testcase for variable-length string
commit 237fee4de158021d51bbf2c0550933f41cd90aab
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-05T00:48:47Z
[FLINK-5734] user proper logger and also add comments
commit 3b38a6f9dee406436864af7c6052db84a9ebfd34
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-05T16:29:16Z
[FLINK-5734] checking endianness should not be involved in generating
procedures for compare function
commit 559daf5d48f1d786de72127c92cb1c1f6a4c2ff9
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-05T20:12:50Z
[FLINK-5734] move byte-operator-mapping to be a constant of class scope
commit 56454f63ebf23ea962c793ac0dc35493bc5629cc
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-05T21:04:17Z
[FLINK-5734] add enable/disable flag in ExecutionConfig
commit 912641c4e3894b9c4cbfe82c2ac86fc96d463725
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-05T22:09:20Z
[FLINK-5734] rename variable names in old compare/swap functions to match
with the new ones
commit bc8bb29d3aec66908aa3e1e91fd5ea2661afd1f8
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-05T22:09:49Z
[FLINK-5734] improve tests
commit 45d0aa590b21d0c23d76601f5e8efe6f17c0211b
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-06T15:14:54Z
[FLINK-5734] add timestamp to generated sorter
commit 3563e213bdb0d54bcee3c64e753f4656862554f3
Author: SerkanAli <erg...@campus.tu-berlin.de>
Date: 2017-03-07T13:27:44Z
[FLINK-5734] RESOURCE_PATH exhibits to a Temporary directory
commit 8d3e08485931c0d0d5bcf6647eb346eff65c6fa7
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-07T22:35:25Z
[FLINK-5734] integrate code generation to flink code
commit 362fc4656f60354e4ec23732ec339067adb47b1e
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-08T00:26:34Z
[FLINK-5734] prevent generating same sorter at the same time
commit 3415cbaf0dc8d9ecaf0f1c9652b15a49eececca2
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-08T00:35:38Z
[FLINK-5734] fix sorting in desc order failed
commit bcf50d285e2040443e1b9857484b63008304c9c5
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-08T12:03:19Z
[FLINK-5734] also cache constructor to save cooking time
commit bc2a37c4b6e41428f320efc9fd70e3b47f6aad5e
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-08T14:34:25Z
[FLINK-5734] refactor integration tests
commit f06f46873587b21357a86897ca5096a0dcb06a20
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-03-08T23:09:08Z
[FLINK-5734] add benchmarking code
commit 7c8f82f3c11bfba3a20b34fe5315a9d0cb827805
Author: SerkanAli <erg...@campus.tu-berlin.de>
Date: 2017-03-09T12:24:52Z
[FLINK-5734] get temporary directory for generated code from task config.
---
If your proj
[GitHub] flink pull request #3393: [FLINK-3903][docs] adding alternative installation...
Github user heytitle commented on a diff in the pull request:
https://github.com/apache/flink/pull/3393#discussion_r102675770
--- Diff: docs/quickstart/setup_quickstart.md ---
@@ -72,6 +72,15 @@ $ cd build-target # this is where Flink is
installed to
~~~
{% endif %}
+### Alternatively
--- End diff --
Should we also put to OSX installation into the if/else?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3393: [FLINK-3903][docs] adding alternative installation...
GitHub user heytitle opened a pull request:
https://github.com/apache/flink/pull/3393
[FLINK-3903][docs] adding alternative installation documentation for â¦
â¦OSX users
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/heytitle/flink
FLINK-3903-homebrew-installation
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3393.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3393
commit 0539e3f72e19ba5523410b4b03bd446969c8fbb7
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-02-22T16:04:42Z
[FLINK-3903][docs] adding alternative installation documentation for OSX
users
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3080: [FLINK-4920] Add a Scala Function Gauge
Github user heytitle commented on a diff in the pull request:
https://github.com/apache/flink/pull/3080#discussion_r95671339
--- Diff:
flink-scala/src/main/scala/org/apache/flink/api/scala/metrics/ScalaGauge.scala
---
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.metrics
+
+import org.apache.flink.metrics.Gauge
+
--- End diff --
IMHO, this class is quite small and obvious what it is doing.
However, if you think it it better to have some explanation, here is my
idea.
```
A ScalaGauge is similar to a Gauge in {@link Metric} but made for Scala API.
```
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3080: [FLINK-4920] Add a Scala Function Gauge
Github user heytitle commented on a diff in the pull request:
https://github.com/apache/flink/pull/3080#discussion_r95666996
--- Diff:
flink-scala/src/test/scala/org/apache/flink/api/scala/metrics/ScalaGaugeTest.scala
---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.metrics
+
+import org.apache.flink.metrics.Gauge
+import org.apache.flink.runtime.metrics.{MetricRegistry,
MetricRegistryConfiguration}
+import org.apache.flink.runtime.metrics.groups.GenericMetricGroup
+import org.apache.flink.util.TestLogger
+import org.junit.Test
+import org.scalatest.junit.JUnitSuiteLike
+
+class ScalaGaugeTest extends TestLogger with JUnitSuiteLike {
+
+ @Test
+ def testGaugeCorrectValue(): Unit = {
+val myGauge = ScalaGauge[Long]( () => 4 )
+assert( myGauge.getValue == 4 )
+ }
+
+ @Test
+ def testRegister(): Unit = {
+val conf =
MetricRegistryConfiguration.defaultMetricRegistryConfiguration()
+val registry = new MetricRegistry(conf)
+val metricGroup = new GenericMetricGroup(registry, null, "world")
+
+val myGauge = ScalaGauge[Long]( () => 4 )
+val gauge = metricGroup.gauge[Long, Gauge[Long]]("max", myGauge);
+
+assert( gauge == myGauge )
--- End diff --
yeah, that's true. I'll remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3080: [FLINK-4920] Add a Scala Function Gauge
Github user heytitle commented on a diff in the pull request:
https://github.com/apache/flink/pull/3080#discussion_r95347426
--- Diff:
flink-scala/src/main/scala/org/apache/flink/api/scala/metrics/ScalaGauge.scala
---
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.metrics
+
+import org.apache.flink.metrics.Gauge
+
+class ScalaGauge[T](value : T) extends Gauge[T] {
--- End diff --
@KurtYoung Thank for the comment. I see your point. I'll fix it
accordingly.
Also, I have an idea of having a companion object of `ScalaGauge`. As a
result, we can instantiate `ScalaGauge` without calling `new`.
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3080: [FLINK-4920] Add a Scala Function Gauge
GitHub user heytitle opened a pull request:
https://github.com/apache/flink/pull/3080
[FLINK-4920] Add a Scala Function Gauge
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build [](https://travis-ci.org/heytitle/flink)
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/heytitle/flink scala-gauge
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3080.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3080
commit 1a464e2b74930c63aada3dbae40b74822ccea89c
Author: heytitle <pat.chor...@gmail.com>
Date: 2016-12-27T22:21:19Z
[FLINK-4920] Introduce Scala Function Gauge
commit aa9b7bb435e97687a8d69820579bddf41b1b29a4
Author: heytitle <pat.chor...@gmail.com>
Date: 2017-01-07T10:30:40Z
[FLINK-4920] Update gauge document
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2611: [FLINK-4778] Update program example in /docs/setup/cli.md...
Github user heytitle commented on the issue: https://github.com/apache/flink/pull/2611 @fhueske you're welcome ð --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2611: Update the way we call WordCount program in setup/...
GitHub user heytitle opened a pull request:
https://github.com/apache/flink/pull/2611
Update the way we call WordCount program in setup/cli.md examples.
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
Due to the change in
https://github.com/apache/flink/commit/0629e25602eefdc239e8e72d9e3c9c1a5164448e,
we need to specify a prefix for `input` and `output` files.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/heytitle/flink update-doc
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2611.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2611
commit f15d38850d1980d7a3e78edf54189483ee880a22
Author: Pattarawat Chormai <pat.chor...@gmail.com>
Date: 2016-10-07T19:34:33Z
Update the way we call WordCount program.
Due to the change in
https://github.com/apache/flink/commit/0629e25602eefdc239e8e72d9e3c9c1a5164448e,
we need to specify a prefix for `input` and `output` files.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
