Hi,
I've gotten an application working with sbt-assembly and spark, thought I'd
present an option. In my experience, trying to bundle any of the Spark
libraries in your uber jar is going to be a major pain. There will be a lot
of deduplication to work through and even if you resolve them it can
Hi Vadim,
After removing provided from org.apache.spark %%
spark-streaming-kinesis-asl I ended up with huge number of deduplicate
errors:
https://gist.github.com/trienism/3d6f8d6b7ff5b7cead6a
It would be nice if you could share some pieces of your mergeStrategy code
for reference.
Also, after
Thanks guys. This might explain why I might be having problems.
Vadim
ᐧ
On Tue, Apr 14, 2015 at 5:27 PM, Mike Trienis mike.trie...@orcsol.com
wrote:
Richard,
You response was very helpful and actually resolved my issue. In case
others run into a similar issue, I followed the procedure:
Have you tried marking only spark-streaming-kinesis-asl as not provided,
and the rest as provided? Then you will not even need to add
kinesis-asl.jar in the spark-submit.
TD
On Tue, Apr 14, 2015 at 2:27 PM, Mike Trienis mike.trie...@orcsol.com
wrote:
Richard,
You response was very helpful
Richard,
You response was very helpful and actually resolved my issue. In case
others run into a similar issue, I followed the procedure:
- Upgraded to spark 1.3.0
- Add all spark related libraries are provided
- Include spark transitive library dependencies
where my build.sbt file
Hi All,
I have having trouble building a fat jar file through sbt-assembly.
[warn] Merging 'META-INF/NOTICE.txt' with strategy 'rename'
[warn] Merging 'META-INF/NOTICE' with strategy 'rename'
[warn] Merging 'META-INF/LICENSE.txt' with strategy 'rename'
[warn] Merging 'META-INF/LICENSE' with
Thanks Vadim, I can certainly consume data from a Kinesis stream when
running locally. I'm currently in the processes of extending my work to a
proper cluster (i.e. using a spark-submit job via uber jar). Feel free to
add me to gmail chat and maybe we can help each other.
On Mon, Apr 13, 2015 at
I don't believe the Kinesis asl should be provided. I used mergeStrategy
successfully to produce an uber jar.
Fyi, I've been having trouble consuming data out of Kinesis with Spark with no
success :(
Would be curious to know if you got it working.
Vadim
On Apr 13, 2015, at 9:36 PM, Mike
Thanks Mike. I was having trouble on EC2.
On Apr 13, 2015, at 10:25 PM, Mike Trienis mike.trie...@orcsol.com wrote:
Thanks Vadim, I can certainly consume data from a Kinesis stream when running
locally. I'm currently in the processes of extending my work to a proper
cluster (i.e. using a
