Hi Jesse,
https://github.com/jbonofre/incubator-beam/tree/DATAFORMAT/sdks/java/extensions/dataformat
it's very simple and stupid and of course not complete at all (I have
other commits but not merged as they need some polishing), but as I
said, it's a base of discussion.
Regards
JB
On 11/29/2016 09:23 PM, Jesse Anderson wrote:
@jb Sounds good. Just let us know once you've pushed.
On Tue, Nov 29, 2016 at 2:54 PM Jean-Baptiste Onofré <j...@nanthrax.net>
wrote:
Good point Eugene.
Right now, it's a DoFn collection to experiment a bit (a pure
extension). It's pretty stupid ;)
But, you are right, depending the direction of such extension, it could
cover more use cases (even if it's not my first intention ;)).
Let me push the branch (pretty small) as an illustration, and in the
mean time, I'm preparing a document (more focused on the use cases).
WDYT ?
Regards
JB
On 11/29/2016 08:47 PM, Eugene Kirpichov wrote:
Hi JB,
Depending on the scope of what you want to ultimately accomplish with
this
extension, I think it may make sense to write a proposal document and
discuss it.
If it's just a collection of utility DoFn's for various well-defined
source/target format pairs, then that's probably not needed, but if it's
anything more, then I think it is.
That will help avoid a lot of churn if people propose reasonable
significant changes.
On Tue, Nov 29, 2016 at 11:15 AM Jean-Baptiste Onofré <j...@nanthrax.net>
wrote:
By the way Jesse, I gonna push my DATAFORMAT branch on my github and I
will post on the dev mailing list when done.
Regards
JB
On 11/29/2016 07:01 PM, Jesse Anderson wrote:
I want to bring this thread back up since we've had time to think about
it
more and make a plan.
I think a format-specific converter will be more time consuming task
than
we originally thought. It'd have to be a writer that takes another
writer
as a parameter.
I think a string converter can be done as a simple transform.
I think we should start with a simple string converter and plan for a
format-specific writer.
What are your thoughts?
Thanks,
Jesse
On Thu, Nov 10, 2016 at 10:33 AM Jesse Anderson <je...@smokinghand.com
wrote:
I was thinking about what the outputs would look like last night. I
realized that more complex formats like JSON and XML may or may not
output
the data in a valid format.
Doing a direct conversion on unbounded collections would work just
fine.
They're self-contained. For writing out bounded collections, that's
where
we'll hit the issues. This changes the uber conversion transform into a
transform that needs to be a writer.
If a transform executes a JSON conversion on a per element basis, we'd
get
this:
{
"key": "value"
}, {
"key": "value"
},
That isn't valid JSON.
The conversion transform would need to know do several things when
writing
out a file. It would need to add brackets for an array. Now we have:
[
{
"key": "value"
}, {
"key": "value"
},
]
We still don't have valid JSON. We have to remove the last comma or
have
the uber transform start putting in the commas, except for the last
element.
[
{
"key": "value"
}, {
"key": "value"
}
]
Only by doing this do we have valid JSON.
I'd argue we'd have a similar issue with XML. Some parsers require a
root
element for everything. The uber transform would have to put the root
element tags at the beginning and end of the file.
On Wed, Nov 9, 2016 at 11:36 PM Manu Zhang <owenzhang1...@gmail.com>
wrote:
I would love to see a lean core and abundant Transforms at the same
time.
Maybe we can look at what Confluent <https://github.com/confluentinc>
does
for kafka-connect. They have official extensions support for JDBC, HDFS
and
ElasticSearch under https://github.com/confluentinc. They put them
along
with other community extensions on
https://www.confluent.io/product/connectors/ for visibility.
Although not a commercial company, can we have a GitHub user like
beam-community to host projects we build around beam but not suitable
for
https://github.com/apache/incubator-beam. In the future, we may have
beam-algebra like http://github.com/twitter/algebird for algebra
operations
and beam-ml / beam-dl for machine learning / deep learning. Also, there
will will be beam related projects elsewhere maintained by other
communities. We can put all of them on the beam-website or like spark
packages as mentioned by Amit.
My $0.02
Manu
On Thu, Nov 10, 2016 at 2:59 AM Kenneth Knowles <k...@google.com.invalid
wrote:
On this point from Amit and Ismaël, I agree: we could benefit from a
place
for miscellaneous non-core helper transformations.
We have sdks/java/extensions but it is organized as separate
artifacts.
I
think that is fine, considering the nature of Join and SortValues. But
for
simpler transforms, Importing one artifact per tiny transform is too
much
overhead. It also seems unlikely that we will have enough commonality
among
the transforms to call the artifact anything other than [some synonym
for]
"miscellaneous".
I wouldn't want to take this too far - even though the SDK many
transforms*
that are not required for the model [1], I like that the SDK artifact
has
everything a user might need in their "getting started" phase of use.
This
user-friendliness (the user doesn't care that ParDo is core and Sum is
not)
plus the difficulty of judging which transforms go where, are probably
why
we have them mostly all in one place.
Models to look at, off the top of my head, include Pig's PiggyBank and
Apex's Malhar. These have different levels of support implied. Others?
Kenn
[1] ApproximateQuantiles, ApproximateUnique, Count, Distinct, Filter,
FlatMapElements, Keys, Latest, MapElements, Max, Mean, Min, Values,
KvSwap,
Partition, Regex, Sample, Sum, Top, Values, WithKeys, WithTimestamps
* at least they are separate classes and not methods on PCollection
:-)
On Wed, Nov 9, 2016 at 6:03 AM, Ismaël Mejía <ieme...@gmail.com>
wrote:
Nice discussion, and thanks Jesse for bringing this subject back.
I agree 100% with Amit and the idea of having a home for those
transforms
that are not core enough to be part of the sdk, but that we all end
up
re-writing somehow.
This is a needed improvement to be more developer friendly, but also
as
a
reference of good practices of Beam development, and for this reason
I
agree with JB that at this moment it would be better for these
transforms
to reside in the Beam repository at least for visibility reasons.
One additional question is if these transforms represent a different
DSL
or
if those could be grouped with the current extensions (e.g. Join and
SortValues) into something more general that we as a community could
maintain, but well even if it is not the case, it would be really
nice
to
start working on something like this.
Ismaël Mejía
On Wed, Nov 9, 2016 at 11:59 AM, Jean-Baptiste Onofré <
j...@nanthrax.net
wrote:
Related to spark-package, we also have Apache Bahir to host
connectors/transforms for Spark and Flink.
IMHO, right now, Beam should host this, not sure if it makes sense
directly in the core.
It reminds me the "Integration" DSL we discussed in the technical
vision
document.
Regards
JB
On 11/09/2016 11:17 AM, Amit Sela wrote:
I think Jesse has a very good point on one hand, while Luke's and
Kenneth's
worries about committing users to specific implementations is in
place.
The Spark community has a 3rd party repository for useful libraries
that
for various reasons are not a part of the Apache Spark project:
https://spark-packages.org/.
Maybe a "common-transformations" package would serve both users
quick
ramp-up and ease-of-use while keeping Beam more "enabling" ?
On Tue, Nov 8, 2016 at 9:03 PM Kenneth Knowles
<k...@google.com.invalid
wrote:
It seems useful for small scale debugging / demoing to have
Dump.toString(). I think it should be named to clearly indicate
its
limited
scope. Maybe other stuff could go in the Dump namespace, but
"Dump.toJson()" would be for humans to read - so it should be
pretty
printed, not treated as a machine-to-machine wire format.
The broader question of representing data in JSON or XML, etc, is
already
the subject of many mature libraries which are already easy to use
with
Beam.
The more esoteric practice of implicit or semi-implicit coercions
seems
like it is also already addressed in many ways elsewhere.
Transform.via(TypeConverter) is basically the same as
MapElements.via(<lambda>) and also easy to use with Beam.
In both of the last cases, there are many reasonable approaches,
and
we
shouldn't commit our users to one of them.
On Tue, Nov 8, 2016 at 10:15 AM, Lukasz Cwik
<lc...@google.com.invalid
wrote:
The suggestions you give seem good except for the the XML cases.
Might want to have the XML be a document per line similar to the
JSON
examples you have been giving.
On Tue, Nov 8, 2016 at 12:00 PM, Jesse Anderson <
je...@smokinghand.com>
wrote:
@lukasz Agreed there would have to be KV handling. I was more
think
that
whatever the addition, it shouldn't just handle KV. It should
handle
Iterables, Lists, Sets, and KVs.
For JSON and XML, I wonder if we'd be able to give someone
something
general purpose enough that you would just end up writing your
own
code
to
handle it anyway.
Here are some ideas on what it could look like with a method and
the
resulting string output:
*Stringify.toJSON()*
With KV:
{"key": "value"}
With Iterables:
["one", "two", "three"]
*Stringify.toXML("rootelement")*
With KV:
<rootelement key=value />
With Iterables:
<rootelement>
<item>one</item>
<item>two</item>
<item>three</item>
</rootelement>
*Stringify.toDelimited(",")*
With KV:
key,value
With Iterables:
one,two,three
Do you think that would strike a good balance between reusable
code
and
writing your own for more difficult formatting?
Thanks,
Jesse
On Tue, Nov 8, 2016 at 11:01 AM Lukasz Cwik
<lc...@google.com.invalid
wrote:
Jesse, I believe if one format gets special treatment in TextIO,
people
will then ask why doesn't JSON, XML, ... also not supported.
Also, the example that you provide is using the fact that the
input
format
is an Iterable<Item>. You had posted a question about using KV
with
TextIO.Write which wouldn't align with the proposed input format
and
still
would require to write a type conversion function, this time
from
KV
to
Iterable<Item> instead of KV to string.
On Tue, Nov 8, 2016 at 9:50 AM, Jesse Anderson <
je...@smokinghand.com>
wrote:
Lukasz,
I don't think you'd need complicated logic for TextIO.Write.
For
CSV
the
call would look like:
Stringify.to("", ",", "\n");
Where the arguments would be Stringify.to(prefix, delimiter,
suffix).
The code would be something like:
StringBuffer buffer = new StringBuffer(prefix);
for (Item item : list) {
buffer.append(item.toString());
if(notLast) {
buffer.append(delimiter);
}
}
buffer.append(suffix);
c.output(buffer.toString());
That would allow you to do the basic CSV, TSV, and other
formats
without
complicated logic. The same sort of thing could be done for
TextIO.Write.
Thanks,
Jesse
On Tue, Nov 8, 2016 at 10:30 AM Lukasz Cwik
<lc...@google.com.invalid
wrote:
The conversion from object to string will have uses outside of
just
TextIO.Write so it seems logical that we would want to have a
ParDo
do
the
conversion.
Text file formats have a lot of variance, even if you consider
the
subset
of CSV like formats where it could have fixed width fields, or
escaping
and
quoting around other fields, or headers that should be placed
at
the
top.
Having all these format conversions within TextIO.Write seems
like
a
lot
of
logic to contain in that transform which should just focus on
writing
to
files.
On Tue, Nov 8, 2016 at 8:15 AM, Jesse Anderson <
je...@smokinghand.com>
wrote:
This is a thread moved over from the user mailing list.
I think there needs to be a way to convert a PCollection<KV>
to
PCollection<String> Conversion.
To do a minimal WordCount, you have to manually convert the
KV
to a
String:
p
.apply(TextIO.Read.from("playing_cards.tsv"))
.apply(Regex.split("\\W+"))
.apply(Count.perElement())
* .apply(MapElements.via((KV<String, Long>
count)
->*
* count.getKey() + ":" +
count.getValue()*
* ).withOutputType(
TypeDescriptors.strings()))*
.apply(TextIO.Write.to
("output/stringcounts"));
This code really should be something like:
p
.apply(TextIO.Read.from("playing_cards.tsv"))
.apply(Regex.split("\\W+"))
.apply(Count.perElement())
* .apply(ToString.stringify())*
.apply(TextIO.Write.to
("output/stringcounts"));
To summarize the discussion:
- JA: Add a method to StringDelegateCoder to output any KV
or
list
- JA and DH: Add a SimpleFunction that takes an type and runs
toString()
on it:
class ToStringFn<InputT> extends SimpleFunction<InputT,
String>
{
public static String apply(InputT input) {
return input.toString();
}
}
- JB: Add a general purpose type converter like in Apache
Camel.
- JA: Add Object support to TextIO.Write that would write out
the
toString of any Object.
My thoughts:
Is converting to a PCollection<String> mostly needed when
you're
using
TextIO.Write? Will a general purpose transform only work in
certain
cases
and you'll normally have to write custom code format the
strings
the
way
you want them?
IMHO, it's yes to both. I'd prefer to add Object support to
TextIO.Write
or
a SimpleFunction that takes a delimiter as an argument.
Making
a
SimpleFunction that's able to specify a delimiter (and
perhaps
a
prefix
and
suffix) should cover the majority of formats and cases.
Thanks,
Jesse
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
