[
https://issues.apache.org/jira/browse/LIVY-368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222678#comment-16222678
]
Devin Boyer commented on LIVY-368:
----------------------------------
This issue still seems to occur in some situations in Livy 0.4. Adding my case
as a comment, but if it seems more appropriate as a separate bug, I can open a
new bug as well.
I have a simple job I submitted as a batch session that adds a partition to a
Hive table:
{{from pyspark.sql import SparkSession
spark = SparkSession.builder.enableHiveSupport().getOrCreate()
spark.sql("ALTER TABLE mytable ADD IF NOT EXISTS PARTITION (partition_info)
location 's3://mydatalocation/on/s3'")
spark.stop()
}}
Without the spark.stop() line, the YARN application continues on forever (state
= RUNNING, looking at the ResourceManager output) and the Livy state for the
batch session stays as 'started'. When I add the spark.stop() line, the
application does get marked as FINISHED by YARN, but Livy still reports the
batch session as 'started'.
> Livy batch state is not being updated properly.
> -----------------------------------------------
>
> Key: LIVY-368
> URL: https://issues.apache.org/jira/browse/LIVY-368
> Project: Livy
> Issue Type: Bug
> Components: Batch
> Affects Versions: 0.3
> Reporter: Sneha Hegde
>
> I tried using an Http client to connect to livy server and work in batch
> mode, it works fine. But I don't see the status being updated properly. It
> always shows the status as "starting" even when it is running in batch mode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
