-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
And to be clear. Yes, execution plans show what exactly it's doing. The
problem is that it's unclear how it's related to the actual Scala/Python
code.
On 7/21/20 15:45, Michal Sankot wrote:
Yes, the problem is that DAGs only refer to code line (action) that
inovked it. It doesn't provide
it's actually
doing.
On 7/21/20 15:36, Russell Spitzer wrote:
Have you looked in the DAG visualization? Each block refer to the code
line invoking it.
For Dataframes the execution plan will let you know explicitly which
operations are in which stages.
On Tue, Jul 21, 2020, 8:18 AM Michal
Hi,
when I analyze and debug our Spark batch jobs executions it's a pain to
find out how blocks in Spark UI Jobs/SQL tab correspond to the actual
Scala code that we write and how much time they take. Would there be a
way to somehow instruct compiler or something and get this information
into
ing took then it would to just do the operation. So most operators
> actually end up measuring the lifetime of the operator which often is the
> time of the entire task minus how long it took for the first task to get to
> that operator. This is also true of WholeStageCodeGen.
> >
&
[
https://issues.apache.org/jira/browse/LIVY-712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michal Sankot closed LIVY-712.
--
Resolution: Workaround
Issue seems to be external - concretely a problem in AWS EMR customizations
[
https://issues.apache.org/jira/browse/LIVY-712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984437#comment-16984437
]
Michal Sankot commented on LIVY-712:
It seems that issue was present in EMR 5.23/5.27 (hadoop libraries
[
https://issues.apache.org/jira/browse/LIVY-712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982583#comment-16982583
]
Michal Sankot commented on LIVY-712:
After further investigation, it seems that problem is unrelated
[
https://issues.apache.org/jira/browse/LIVY-712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982583#comment-16982583
]
Michal Sankot edited comment on LIVY-712 at 11/26/19 3:15 PM:
--
After further
[
https://issues.apache.org/jira/browse/LIVY-712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981471#comment-16981471
]
Michal Sankot commented on LIVY-712:
Sure,
* create a Scala Spark job that throws NullPointerException
Hey,
Some time ago we've setup custom domain SSL access to our GAE application
through Google Apps for Work account as it was the only way to do it.
Now Google has implemented Custom domains SSL directly in Google Cloud
Platform (Announcement
I have Java GAE app with modules. Default front-end module is marked as
automatic-scaling
min-idle-instances1/min-idle-instances/automatic-scaling
however when I check chart of instances for last 24 hours, I see that there
is a period where no instance was running. I would expect that
Hey,
I have a GAE Java project where I use Jersey (1.17) and Guice (3.0).
SessionScoped beans work in local dev, but don't work when deployed on GAE.
The problem is that they don't keep session state.
Sessions are enabled in web.xml: sessions-enabledtrue/sessions-enabled
My Session bean
ok,
I tried to isolate problematic part of code and found the culprit. I was
using two different buffers for saving incomming data, where the forst one
was saving data returned by characters and second appended sone other text
to the contents of first one, after parsing finished. Problem was that
Hi,
I have problem with SAX bit of Xerces. I use SAX to get lines of an element
of specified tag and print them out.
I was using older version of Xerces with which it run fine. When I replaced
old xerces.jar with new xercesImpl.jar SAX starts to behave wierd.
CDATA element content which is
15 matches
Mail list logo
