Thanks guys. This might explain why I might be having problems.
Vadim
ᐧ
On Tue, Apr 14, 2015 at 5:27 PM, Mike Trienis
wrote:
> Richard,
>
> You response was very helpful and actually resolved my issue. In case
> others run into a similar issue, I followed the procedure:
>
>- Upgraded to sp
Have you tried marking only spark-streaming-kinesis-asl as not provided,
and the rest as provided? Then you will not even need to add
kinesis-asl.jar in the spark-submit.
TD
On Tue, Apr 14, 2015 at 2:27 PM, Mike Trienis
wrote:
> Richard,
>
> You response was very helpful and actually resolved m
Richard,
You response was very helpful and actually resolved my issue. In case
others run into a similar issue, I followed the procedure:
- Upgraded to spark 1.3.0
- Add all spark related libraries are "provided"
- Include spark transitive library dependencies
where my build.sbt file
Hi,
I've gotten an application working with sbt-assembly and spark, thought I'd
present an option. In my experience, trying to bundle any of the Spark
libraries in your uber jar is going to be a major pain. There will be a lot
of deduplication to work through and even if you resolve them it can be
Hi Vadim,
After removing "provided" from "org.apache.spark" %%
"spark-streaming-kinesis-asl" I ended up with huge number of deduplicate
errors:
https://gist.github.com/trienism/3d6f8d6b7ff5b7cead6a
It would be nice if you could share some pieces of your mergeStrategy code
for reference.
Also, a
Thanks Mike. I was having trouble on EC2.
> On Apr 13, 2015, at 10:25 PM, Mike Trienis wrote:
>
> Thanks Vadim, I can certainly consume data from a Kinesis stream when running
> locally. I'm currently in the processes of extending my work to a proper
> cluster (i.e. using a spark-submit job vi
Thanks Vadim, I can certainly consume data from a Kinesis stream when
running locally. I'm currently in the processes of extending my work to a
proper cluster (i.e. using a spark-submit job via uber jar). Feel free to
add me to gmail chat and maybe we can help each other.
On Mon, Apr 13, 2015 at 6
I don't believe the Kinesis asl should be provided. I used mergeStrategy
successfully to produce an "uber jar."
Fyi, I've been having trouble consuming data out of Kinesis with Spark with no
success :(
Would be curious to know if you got it working.
Vadim
> On Apr 13, 2015, at 9:36 PM, Mike T
Hi All,
I have having trouble building a fat jar file through sbt-assembly.
[warn] Merging 'META-INF/NOTICE.txt' with strategy 'rename'
[warn] Merging 'META-INF/NOTICE' with strategy 'rename'
[warn] Merging 'META-INF/LICENSE.txt' with strategy 'rename'
[warn] Merging 'META-INF/LICENSE' with strat
