[jira] [Commented] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark
[ https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886016#comment-15886016 ] Robert Kanter commented on OOZIE-2792: -- +1 on the amendment patch. Thanks for catching that. > Hive2 action is not parsing Spark application ID from log file properly when > Hive is on Spark > - > > Key: OOZIE-2792 > URL: https://issues.apache.org/jira/browse/OOZIE-2792 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Xiaobin Zheng >Assignee: Xiaobin Zheng >Priority: Minor > Fix For: 5.0.0 > > Attachments: OOZIE-2792-1.patch, OOZIE-2792-2.patch, > OOZIE-2792-3.patch, OOZIE-2792-4-amendment.patch, OOZIE-2792-4.patch > > > When Hive2 is on Spark, hive action is not able to parse Spark application ID > from log file as 'externalChildID' like Spark/MR actions. This makes it hard > to tell which job hive launches from Oozie server for a particular workflow. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark
[
https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883888#comment-15883888
]
Hadoop QA commented on OOZIE-2792:
--
Testing JIRA OOZIE-2792
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1886
.Tests failed: 1
.Tests errors: 0
.The patch failed the following testcases:
. testFsFailover(org.apache.oozie.action.TestActionFailover)
.Tests failing with errors:
.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3660/
> Hive2 action is not parsing Spark application ID from log file properly when
> Hive is on Spark
> -
>
> Key: OOZIE-2792
> URL: https://issues.apache.org/jira/browse/OOZIE-2792
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Xiaobin Zheng
>Assignee: Xiaobin Zheng
>Priority: Minor
> Fix For: 5.0.0
>
> Attachments: OOZIE-2792-1.patch, OOZIE-2792-2.patch,
> OOZIE-2792-3.patch, OOZIE-2792-4-amendment.patch, OOZIE-2792-4.patch
>
>
> When Hive2 is on Spark, hive action is not able to parse Spark application ID
> from log file as 'externalChildID' like Spark/MR actions. This makes it hard
> to tell which job hive launches from Oozie server for a particular workflow.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark
[
https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866983#comment-15866983
]
Hadoop QA commented on OOZIE-2792:
--
Testing JIRA OOZIE-2792
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 6 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1879
.Tests failed: 0
.Tests errors: 5
.The patch failed the following testcases:
.
.Tests failing with errors:
. testJobIDPattern(org.apache.oozie.action.hadoop.TestHiveMain)
. testMain(org.apache.oozie.action.hadoop.TestHiveMain)
.
testActionConfLoadDefaultResources(org.apache.oozie.action.hadoop.TestHiveActionExecutor)
. testSetupMethods(org.apache.oozie.action.hadoop.TestHiveActionExecutor)
. testHiveAction(org.apache.oozie.action.hadoop.TestHiveActionExecutor)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3649/
> Hive2 action is not parsing Spark application ID from log file properly when
> Hive is on Spark
> -
>
> Key: OOZIE-2792
> URL: https://issues.apache.org/jira/browse/OOZIE-2792
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Xiaobin Zheng
>Assignee: Xiaobin Zheng
>Priority: Minor
> Attachments: OOZIE-2792-1.patch, OOZIE-2792-2.patch,
> OOZIE-2792-3.patch, OOZIE-2792-4.patch
>
>
> When Hive2 is on Spark, hive action is not able to parse Spark application ID
> from log file as 'externalChildID' like Spark/MR actions. This makes it hard
> to tell which job hive launches from Oozie server for a particular workflow.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark
[
https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862610#comment-15862610
]
Hadoop QA commented on OOZIE-2792:
--
Testing JIRA OOZIE-2792
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 8 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3645/
> Hive2 action is not parsing Spark application ID from log file properly when
> Hive is on Spark
> -
>
> Key: OOZIE-2792
> URL: https://issues.apache.org/jira/browse/OOZIE-2792
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Xiaobin Zheng
>Assignee: Xiaobin Zheng
>Priority: Minor
> Attachments: OOZIE-2792-1.patch, OOZIE-2792-2.patch,
> OOZIE-2792-3.patch
>
>
> When Hive2 is on Spark, hive action is not able to parse Spark application ID
> from log file as 'externalChildID' like Spark/MR actions. This makes it hard
> to tell which job hive launches from Oozie server for a particular workflow.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark
[
https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860557#comment-15860557
]
Robert Kanter commented on OOZIE-2792:
--
This looks good.
I know this is a little outside the scope of this patch, but it's kind of
related and should be pretty quick to do: can you add a similar test for the
other Action types that have a jobid pattern (e.g.
{{PigMain.PIG_JOB_IDS_PATTERNS}}, etc)? That's probably something we should
have already had tests for.
> Hive2 action is not parsing Spark application ID from log file properly when
> Hive is on Spark
> -
>
> Key: OOZIE-2792
> URL: https://issues.apache.org/jira/browse/OOZIE-2792
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Xiaobin Zheng
>Assignee: Xiaobin Zheng
>Priority: Minor
> Attachments: OOZIE-2792-1.patch
>
>
> When Hive2 is on Spark, hive action is not able to parse Spark application ID
> from log file as 'externalChildID' like Spark/MR actions. This makes it hard
> to tell which job hive launches from Oozie server for a particular workflow.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark
[ https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859261#comment-15859261 ] Peter Cseh commented on OOZIE-2792: --- Thanks for the contribution [~zhengxb2005]! Tests are passing locally. +1 (nonbinding) for the patch. > Hive2 action is not parsing Spark application ID from log file properly when > Hive is on Spark > - > > Key: OOZIE-2792 > URL: https://issues.apache.org/jira/browse/OOZIE-2792 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Xiaobin Zheng >Assignee: Xiaobin Zheng >Priority: Minor > Attachments: OOZIE-2792-1.patch > > > When Hive2 is on Spark, hive action is not able to parse Spark application ID > from log file as 'externalChildID' like Spark/MR actions. This makes it hard > to tell which job hive launches from Oozie server for a particular workflow. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark
[ https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858422#comment-15858422 ] Xiaobin Zheng commented on OOZIE-2792: -- The above failure testBundleRerunInPausedWithError failed in other trunk builds as well. It is unrelated to this patch. [~rkanter] Can you pls help review this patch? This is tested in a real cluster as well. 'externalChildIDs' field for hive2 action are displaying the Spark application ID properly with this patch. > Hive2 action is not parsing Spark application ID from log file properly when > Hive is on Spark > - > > Key: OOZIE-2792 > URL: https://issues.apache.org/jira/browse/OOZIE-2792 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Xiaobin Zheng >Assignee: Xiaobin Zheng >Priority: Minor > Attachments: OOZIE-2792-1.patch > > > When Hive2 is on Spark, hive action is not able to parse Spark application ID > from log file as 'externalChildID' like Spark/MR actions. This makes it hard > to tell which job hive launches from Oozie server for a particular workflow. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2792) Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark
[
https://issues.apache.org/jira/browse/OOZIE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857834#comment-15857834
]
Hadoop QA commented on OOZIE-2792:
--
Testing JIRA OOZIE-2792
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [1] new bugs found below threshold in total that
must be fixed.
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:red}-1{color} There are [1] new bugs found below threshold in
[sharelib/hive2] that must be fixed.
.You can find the FindBugs diff here (look for the red and orange ones):
sharelib/hive2/findbugs-new.html
.The most important FindBugs errors are:
.At Hive2Main.java:[line 44]:
org.apache.oozie.action.hadoop.Hive2Main.HIVE2_JOB_IDS_PATTERNS should be
package protected
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in
[hadooplibs/hadoop-utils-2].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1873
.Tests failed: 1
.Tests errors: 0
.The patch failed the following testcases:
.
testBundleRerunInPausedWithError(org.apache.oozie.command.bundle.TestBundleRerunXCommand)
.Tests failing with errors:
.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3633/
> Hive2 action is not parsing Spark application ID from log file properly when
> Hive is on Spark
> -
>
> Key: OOZIE-2792
> URL: https://issues.apache.org/jira/browse/OOZIE-2792
> Project: Oozie
> Issue Type: Bug
> Components: action
>Reporter: Xiaobin Zheng
>Assignee: Xiaobin Zheng
>Priority: Minor
> Attachments: OOZIE-2792-1.patch
>
>
> When Hive2 is on Spark, hive action is not able to parse Spark application ID
> from log file as 'externalChildID' like Spark/MR actions. This makes it hard
> to tell which job hive launches from Oozie server for a particular workflow.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
