re you using spark standalone, yarn, or mesos?
>>
>>
>> Thank You,
>>
>> Irving Duran
>>
>> On Thu, Dec 22, 2016 at 1:42 AM, Tal Grynbaum <tal.grynb...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have a dataframe that
Hi,
I have a dataframe that contain an embedded json string in one of the fields
I'd tried to write a UDF function that will parse it using lift-json, but
it seems to take a very long time to process, and it seems that only the
master node is working.
Has anyone dealt with such a scenario before
You need to download all the dependencies of that jar as well
On Mon, Sep 5, 2016, 06:59 Divya Gehlot wrote:
> Hi,
> I am using spark-csv to parse my input files .
> If I use --package option it works fine but if I download
>
My guess is that you're running out of memory somewhere. Try to increase
the driver memory and/or executor memory.
On Sat, Sep 3, 2016, 11:42 kant kodali wrote:
> I am running this on aws.
>
>
>
> On Fri, Sep 2, 2016 11:49 PM, kant kodali kanth...@gmail.com wrote:
>
>> I am
ther words, even if spark was
rewritten in python, and was to focus on python only, you would still not
get those features.
--
*Tal Grynbaum* / *CTO & co-founder*
m# +972-54-7875797
mobile retention done right
if one of you can think of a better solution.
Thanks
Tal
--
*Tal Grynbaum* / *CTO & co-founder*
m# +972-54-7875797
mobile retention done right
Did you specify -Dscala-2.10
As in
./dev/change-scala-version.sh 2.10 ./build/mvn -Pyarn -Phadoop-2.4
-Dscala-2.10 -DskipTests clean package
If you're building with scala 2.10
On Sat, Aug 27, 2016, 00:18 Marco Mistroni wrote:
> Hello Michael
> uhm i celebrated too soon
dir when jobs fail.
> In the previous versions of Spark, there was a way to directly write data
> in a destination though,
> Spark v2.0+ has no way to do that because of the critial issue on S3 (See:
> SPARK-10063).
>
> // maropu
>
>
> On Thu, Aug 25, 2016 at 2:40 PM, Tal Grynbaum
I read somewhere that its because s3 has to know the size of the file
upfront
I dont really understand this, as to why is it ok not to know it for the
temp files and not ok for the final files.
The delete permission is the minor disadvantage from my side, the worst
thing is that i have a
You can use rank window function to rank each row in the group, and then
filter the rowz with rank < 50
On Wed, Jul 6, 2016, 14:07 wrote:
> hi there
> I have a DF with 3 columns: id , pv, location.(the rows are already
> grouped by location and sort by pv in des) I wanna
10 matches
Mail list logo