Re: Debug spark jobs on Intellij

2016-05-31 Thread Marcelo Oikawa
EachRemaining(message -> { > //breakpoint doenst stop here > > }) > }); > > *toDebug.first* // now is when this method will run > foreachPartition is a void method. > > > 2016-05-31 17:59 GMT-03:00 Marcelo Oikawa <marcelo.oik...@webradar.com>: > >&

Re: Debug spark jobs on Intellij

2016-05-31 Thread Marcelo Oikawa
oblem itself because the code inside forEachRemaining runs well but I can't debug this block. > This is when the spark will run the operations. > Have you tried that? > > Cheers. > > 2016-05-31 17:18 GMT-03:00 Marcelo Oikawa <marcelo.oik...@webradar.com>: > >> He

Debug spark jobs on Intellij

2016-05-31 Thread Marcelo Oikawa
Hello, list. I'm trying to debug my spark application on Intellij IDE. Before I submit my job, I ran the command line: export SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=4000 after that: bin/spark-submit app-jar-with-dependencies.jar The IDE connects with

Re: Problem with jackson lib running on spark

2016-04-01 Thread Marcelo Oikawa
ting the same error. Seems strange because > when I see the generated jar there is nothing related as AnnotationMap but > there is a databind there. > > > ​ > > >> >> On Thu, Mar 31, 2016 at 11:37 AM, Marcelo Oikawa < >> marcelo.oik...@webradar.com> wrote: >&g

Re: Problem with jackson lib running on spark

2016-03-31 Thread Marcelo Oikawa
On Thu, Mar 31, 2016 at 11:37 AM, Marcelo Oikawa < > marcelo.oik...@webradar.com> wrote: > >> Hi, Alonso. >> >> As you can see jackson-core is provided by several libraries, try to >>> exclude it from spark-core, i think the minor version is included within &g

Re: Problem with jackson lib running on spark

2016-03-31 Thread Marcelo Oikawa
, then programming > must be the process of putting ..." > - Edsger Dijkstra > > "If you pay peanuts you get monkeys" > > > 2016-03-31 20:01 GMT+02:00 Marcelo Oikawa <marcelo.oik...@webradar.com>: > >> Hey, Alonso. >> >> here is the outp

Re: Problem with jackson lib running on spark

2016-03-31 Thread Marcelo Oikawa
ebe ser el proceso de introducirlos..." > - Edsger Dijkstra > > My favorite quotes (today): > "If debugging is the process of removing software bugs, then programming > must be the process of putting ..." > - Edsger Dijkstra > > "If you pay peanuts you get monkey

Re: Problem with jackson lib running on spark

2016-03-31 Thread Marcelo Oikawa
.1 provided and the jackson version 2.4.4 was not listed in maven dependencies... > > Consider using maven-shade-plugin to resolve the conflict if you use maven. > > Cheers > > On Thu, Mar 31, 2016 at 9:50 AM, Marcelo Oikawa < > marcelo.oik...@webradar.com> wrote: >

Problem with jackson lib running on spark

2016-03-31 Thread Marcelo Oikawa
Hi, list. We are working on a spark application that sends messages to Druid. For that, we're using Tranquility core. In my local test, I'm using the "spark-1.6.1-bin-hadoop2.6" distribution and the following dependencies in my app: org.apache.spark spark-streaming_2.10 1.6.1