GitHub user aljoscha opened a pull request:
https://github.com/apache/incubator-flink/pull/119
Bring Scala API up to speed with Java API
First of all, don't try reading central Scala API rewrite commit. It's
basically a rewrite from Scratch, so the diff won't make a lot of sense. Just
look at the result.
The commits before the central commit massage the rest of the system into
shape to allow the rewrite. The commits after change some small things and add
examples and ITCases.
@mbalassi: How's that for working on Sundays... :smile_cat:
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/incubator-flink scala-rework
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-flink/pull/119.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #119
----
commit eae0e1bf3dc1cd34a51efdae835be2bb3c5a53e8
Author: Aljoscha Krettek <[email protected]>
Date: 2014-08-22T10:24:11Z
Refactor TupleTypeInfo and add GenericPairComparator
Now we have TupleTypeInfoBase, TupleSerializerBase, and
TupleComparatorBase. They
are now super classes of TupleTypeInfo and the others.
Also rename compare on DataInputView to compareSerialized because Scala
cannot distinguish between the to compare methods for some reason.
This change is necessary for allowing the Scala API to reuse most of the
functionality.
The GenericPairComparator uses the new extractKeys method of
TypeComparator to compare values of any type. This replaces
TuplePairComparator and some other special-case pair comparators. This
is preparatory work for enabling support for Scala Tuples and POJO
comparators.
commit 320e7f84e004c7aecb7edc39293b910c4205e974
Author: Aljoscha Krettek <[email protected]>
Date: 2014-08-22T16:17:48Z
Perform TypeExtraction outside of Java Operators
Before, FlatMapOperator, GroupReduceOperator, MapOperator, and
MapPartitionOperator
performed the Type extraction themselves while the other Operators had
TypeInformation
parameters. Now the are all unified, which makes it possible to use them
from the
Scala API.
Also Key extraction for selector functions is moved outside of Keys.java
commit 998050fa17fe7466818ef9b3830b3fbeccdeec1f
Author: Aljoscha Krettek <[email protected]>
Date: 2014-08-26T14:15:05Z
Move the Iteration "Operators" to the operators package
I'm doing this in preparation for the Scala API rework, since then I can
just copy over
the operators package and use them for Scala.
commit 35425515373313640cc85c0b697d3a71754a242d
Author: Aljoscha Krettek <[email protected]>
Date: 2014-08-26T15:22:18Z
Make DistinctOperator and Keys use TupleComparatorBase
Was TupleComparator before which does not work when used from the Scala API.
commit da280502192e555e660ad04797d8533648d2681c
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-02T16:40:17Z
Change ObjectArrayTypeInfo.getInfoFor to use componentInfo.isTupleType()
This is necessary because the previous test didn't work for Scala tuples.
commit c20e97534a28db6e5d4ff710c419575d0d1b6b6e
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-02T16:41:53Z
Refactor Tuple wrapping/unwrapping in Join and CoGroup
Previously both sides were always wrapped/unwrapped, even if one side
was a tuple type. This was not compatible with the Scala tuples and is
also not necessary anymore because of the GenericPairComparator.
commit cbbde178ad98f72283d18e57eb7b151f6dfb821e
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-03T12:50:42Z
Move RichFunctions to api.common package
They were in api.java before but they can be used from Scala,
just like the regular functions.
commit 216479be4b9f4a4ac554df99d16029e12e3caadd
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-04T18:51:43Z
Rewrite the Scala API as (somewhat) thin Layer on Java API
Don't bother looking at the diff, this is almost a complete rewrite of
the previous Scala API. This uses all the work put into the Java API,
such as TypeInformation, the serializers and comparators and the
operators.
The Scala DataSet and ExecutionEnvironment wrap their respective Java
equivalents. TypeInformation is generated by a macro that uses
TypeInformationGen and other macro support classes. The Java
TypeExtractor is completely bypassed but the TypeInformation and
sub-classes are created by the Scala type analyzer. There is special
support for Scala Tuples in the form of ScalaTupleTypeInfo,
ScalaTupleSerializer, and ScalaTupleComparator.
This also adds tests to flink-scala that are ports of the tests in
flink-java.
There are not yet any Scala specific tests in flink-tests. All the
scala example ITCases are commented out, as well as the examples
themselves. Those will be uncommented once the examples are ported. This
will happen in separate commits.
commit f1697dc0f53677c801734b7c23b5647f58f11954
Author: vasia <[email protected]>
Date: 2014-09-09T09:16:59Z
Connected Components example Scala API rewrite
commit 2693d0418270d253be4439edcbdd9a88ff47ab59
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-09T12:48:27Z
Add ConnectedComponentsITCase for Scala Example
Also fix some formatting in the example code
commit 99ef2b8b101212c650f1002dde709811bfb69a78
Author: Kostas Tzoumas <[email protected]>
Date: 2014-09-09T14:45:01Z
WebLog Analysis example
commit cbca37dc6252aaa0845c7ed663b1d9ead20dfa82
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-09T15:52:12Z
Add WebLogAnalysisITCase for Scala Example
Some minor reformatting of example code and scaladoc.
commit 381bb3295bc9cd0b244054143f471361e09d608a
Author: Kostas Tzoumas <[email protected]>
Date: 2014-09-10T14:20:46Z
Transitive closure Scala example
commit a163c7b814c8cab632def36f74fc1b0d40af456f
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-10T15:59:55Z
Fix formatting in TransitiveClosureNaive Scala Example
Also actually use termination criterion in TransitivelClosureNaive
Java example.
commit 647cc2c9b962264a36e895a89a30938328a0166b
Author: Fabian Hueske <[email protected]>
Date: 2014-09-05T22:59:44Z
Added TriangleEnumeration examples for reworked Scala API.
commit 7a236c29d202a5bcc41078313b37d83b24ff1dc7
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-11T09:14:48Z
Add ITCases for TriangleEnumeration Scala Examples
Also fix some formatting and make TriangleEnumerationOpt Scala produce the
same output as the Java version.
commit 9bf4fc0ec37eed51df595a321e62bd092fa236cb
Author: Fabian Hueske <[email protected]>
Date: 2014-09-11T13:42:29Z
Adds PageRankBasic Scala example. Removes old Scala examples
commit a97f2076bcafcacd7e8d673e97344dd3bf342f60
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-11T14:14:32Z
Add logback-test.xml to java8-tests and logback.xml to java-examples
commit 1f24e6bc31b420858c047d2b066239e300c58df2
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-11T15:22:29Z
Add PageRankITCase for Scala Example
Also fix formatting in PageRank Scala Example.
commit 97115f173e4703a9ba0746d9149ba9e190ef921f
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-11T15:30:50Z
Fix formatting in EnumTriangles Scala Examples
commit 34989957131b5ce1c6b857dabbc6640301a792d5
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-11T15:47:08Z
Fix formatting in some Scala Examples
Link to ExampleData was not fully qualified.
commit b4b9b742f9832cea30a5dee68557267cd28a6245
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-11T15:54:08Z
Remove Old/Deprecated Scala Examples and ITCases
commit 167cd646abd0248d8e47ef26f03f1426e97a043f
Author: Kostas Tzoumas <[email protected]>
Date: 2014-09-11T16:25:40Z
Pi estimation example job in Scala
commit b68438630f583a7f77604d3b44368b0d6fca72ea
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-11T16:47:58Z
Embellish comments of PiEstimation Scala Example
commit 12ebcf9801b0cda98f4f344c77b0e8c5e7ca903a
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-12T09:38:20Z
[scala] Add Field Name Key Support for Scala Case Classes
This does not change the runtime behavious. The key field names are
mapped to tuple indices at pre-flight time.
Also extends tests to cover the new feature.
commit d4cfb231554e8f9aedbb42ce8c27421bc83520b3
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-12T13:50:09Z
Fix typo, getSematicProperties -> getSemanticProperties
commit 183b4b9e1c87fcc0afd111c39ffa5e608ecb208b
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-12T14:25:56Z
Add SemanticPropertiesTranslationTest for Scala API
Fix join operator to user proper EquiJoin constructor so that semantic
properties are actually retrieved.
commit bb0e263977cd43aad5e9c2eb348dc67a837a55a2
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-12T14:26:38Z
Add Scala API completeness Test
This is very naive right now, just checks whether a function of the same
name is available. There are lists and regex patterns of excluded
functions since not all functions in the Java API need to be available
in the Scala API.
commit c2b25cc755be9ce6eda3311fda29326b4523fa1b
Author: Aljoscha Krettek <[email protected]>
Date: 2014-09-14T16:25:46Z
Fix formatting in EnumTrianglesBasic.scala
commit 43425b7599c5bdbf611b92e4de08e5f5feec3cbc
Author: mbalassi <[email protected]>
Date: 2014-09-13T13:15:17Z
Added LinearRegression scala example. Removed old BGD example.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
