Thank you Amit and JB!
This is not related to DC/OS itself, but I ran into a problem when launching a
Spark job on a cluster with spark-submit. My Spark job written in Beam can’t
read the specified gs file. I got the following error:
Caused by: java.io.IOException: Unable to find handler for
gs://beam-samples/sample.txt
at
org.apache.beam.sdk.util.IOChannelUtils.getFactory(IOChannelUtils.java:307)
at
org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:528)
at
org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:271)
at
org.apache.beam.runners.spark.io.SourceRDD$Bounded$1.hasNext(SourceRDD.java:125)
Then I thought about switching to reading from another source, but I saw in
Beam’s documentation that TextIO can only read from files in Google Cloud
Storage (prefixed with gs://) when running in cluster mode. How do you guys
doing file IO in Beam when using the SparkRunner?
Thank you,
Chaoran
> On Jan 22, 2017, at 4:32 AM, Amit Sela <amitsel...@gmail.com> wrote:
>
> I'lll join JB's comment on the Spark runner saying that submitting Beam
> pipelines using the Spark runner can be done using Spark's spark-submit
> script, find out more in the Spark runner documentation
> <https://beam.apache.org/documentation/runners/spark/>.
>
> Amit.
>
> On Sun, Jan 22, 2017 at 8:03 AM Jean-Baptiste Onofré <j...@nanthrax.net
> <mailto:j...@nanthrax.net>> wrote:
> Hi,
>
> Not directly DCOS (I think Stephen did some test on it), but I have a
> platform running Spark and Flink with Beam on Mesos + Marathon.
>
> It basically doesn't have anything special as running piplines uses
> spark-submit (as on in Spark "natively").
>
> Regards
> JB
>
> On 01/22/2017 12:56 AM, Chaoran Yu wrote:
> > Hello all,
> >
> > Has anyone had experience using Beam on DC/OS? I want to run Beam code
> >
> > executed with Spark runner on DC/OS. As a next step, I would like to run the
> >
> > Flink runner as well. There doesn't seem to exist any information
> > about running
> >
> > Beam on DC/OS I can find on the web. So some pointers are greatly
> > appreciated.
> >
> > Thank you,
> >
> > Chaoran Yu
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org <mailto:jbono...@apache.org>
> http://blog.nanthrax.net <http://blog.nanthrax.net/>
> Talend - http://www.talend.com <http://www.talend.com/>
