[jira] [Updated] (PIG-5468) Remove Google Analytics from the Pig Website
[
https://issues.apache.org/jira/browse/PIG-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niall Pemberton updated PIG-5468:
-
Description:
Hi Pig Team
The ASF {_}*Privacy Policy*{_}[1][2] does not permit the use of _*Google
Analytics*_ on any ASF websites and the ASF Infra team will soon enforce a
{_}*Content Security Policy*{_}(CSP) that will block access to external
trackers:
* [https://lists.apache.org/thread/w34sd92v4rz3j28hyddmt5tbprbdq6lc]
Please could you remove the use of the Google Analytics from the Pig website?
* [https://lists.apache.org/thread/4I
17v5034773lhoytoqs7h343vpjnwn1c|https://lists.apache.org/thread/417v5034773lhoytoqs7h343vpjnwn1c]
I would have submitted a patch to remove Google Analytics, but it seems that
Pig is pulling in file _*site-to-xhtml.xsl*_ from the old Hadoop Subversion
repository, via an svn:external link
*
[https://svn.apache.org/viewvc/hadoop/common/site/main/author/src/documentation/skins/hadoop-pelt/xslt/html/site-to-xhtml.xsl?view=markup]
* [https://svn.apache.org/viewvc/pig/site/author/src/documentation/]
Since Hadoop has now moved to GitHub, then it would probably be a good idea to
copy that folder (as its not longer maintained) into your site and maintain it
within the Pig project - or ask the hadoop project to modify it
The ASF hosts its own _*Matomo*_ instance to provide projects with analytics
and you can request a tracking id for your project by sending a mail to
*privacy AT apache.org.*
*
[https://privacy.apache.org/faq/committers.html#can-i-use-web-analytics-matomo]
Additionally I would recommend reviewing any external resources loaded by your
website. The Content Security Policy will prevent any resources being loaded
from 3rd Party providers that the ASF does not have a Data Processing Agreement
(DPA) with. On the 1st February Infra will begin a temporary "brownout" when
the CSP will be turned on for a short period. This will allow projects to check
which parts, if any, of their websites will stop working. The Privacy FAQ
answers a number of questions about which external providers are permitted or
not:
* [https://privacy.apache.org/faq/committers.html]
Thanks
Niall
[1] [https://privacy.apache.org/policies/website-policy.html]
[2] [https://privacy.apache.org/faq/committers.html#can-i-use-google-analytics]
> Remove Google Analytics from the Pig Website
>
>
> Key: PIG-5468
> URL: https://issues.apache.org/jira/browse/PIG-5468
> Project: Pig
> Issue Type: Task
>Reporter: Niall Pemberton
>Priority: Major
>
> Hi Pig Team
> The ASF {_}*Privacy Policy*{_}[1][2] does not permit the use of _*Google
> Analytics*_ on any ASF websites and the ASF Infra team will soon enforce a
> {_}*Content Security Policy*{_}(CSP) that will block access to external
> trackers:
> * [https://lists.apache.org/thread/w34sd92v4rz3j28hyddmt5tbprbdq6lc]
> Please could you remove the use of the Google Analytics from the Pig website?
> * [https://lists.apache.org/thread/4I
> 17v5034773lhoytoqs7h343vpjnwn1c|https://lists.apache.org/thread/417v5034773lhoytoqs7h343vpjnwn1c]
> I would have submitted a patch to remove Google Analytics, but it seems that
> Pig is pulling in file _*site-to-xhtml.xsl*_ from the old Hadoop Subversion
> repository, via an svn:external link
> *
> [https://svn.apache.org/viewvc/hadoop/common/site/main/author/src/documentation/skins/hadoop-pelt/xslt/html/site-to-xhtml.xsl?view=markup]
> * [https://svn.apache.org/viewvc/pig/site/author/src/documentation/]
> Since Hadoop has now moved to GitHub, then it would probably be a good idea
> to copy that folder (as its not longer maintained) into your site and
> maintain it within the Pig project - or ask the hadoop project to modify it
>
> The ASF hosts its own _*Matomo*_ instance to provide projects with analytics
> and you can request a tracking id for your project by sending a mail to
> *privacy AT apache.org.*
> *
> [https://privacy.apache.org/faq/committers.html#can-i-use-web-analytics-matomo]
> Additionally I would recommend reviewing any external resources loaded by
> your website. The Content Security Policy will prevent any resources being
> loaded from 3rd Party providers that the ASF does not have a Data Processing
> Agreement (DPA) with. On the 1st February Infra will begin a temporary
> "brownout" when the CSP will be turned on for a short period. This will allow
> projects to check which parts, if any, of their websites will stop working.
> The Privacy FAQ answers a number of questions about which external providers
> are permitted or not:
> * [https://privacy.apache.org/faq/committers.html]
> Thanks
>
[jira] [Updated] (PIG-5468) Remove Google Analytics from the Pig Website
[ https://issues.apache.org/jira/browse/PIG-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niall Pemberton updated PIG-5468: - Summary: Remove Google Analytics from the Pig Website (was: Remove Google Analytics from the Juneau Website) > Remove Google Analytics from the Pig Website > > > Key: PIG-5468 > URL: https://issues.apache.org/jira/browse/PIG-5468 > Project: Pig > Issue Type: Task >Reporter: Niall Pemberton >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (PIG-5468) Remove Google Analytics from the Juneau Website
Niall Pemberton created PIG-5468: Summary: Remove Google Analytics from the Juneau Website Key: PIG-5468 URL: https://issues.apache.org/jira/browse/PIG-5468 Project: Pig Issue Type: Task Reporter: Niall Pemberton -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5467) Providing a Docker Image for Apache Pig
[
https://issues.apache.org/jira/browse/PIG-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qiheng He updated PIG-5467:
---
Description:
- I noticed that [https://pig.apache.org/docs/r0.17.0/start.html] requires
users to manually download JDK 7, Hadoop 2.x, and Apache Pig itself.
- But most developers run higher versions of JDK on their development machines
through *SDKMAN!* and {*}Docker CE{*}, and JDK 7 is EOL. If Apache Pig can
provide a Docker Image, it will obviously help with quick testing.
- This would help write unit tests on third-party libraries like
*testcontainers-java* .
was:
- I noticed that https://pig.apache.org/docs/r0.17.0/start.html requires users
to manually download JDK 7, Hadoop 2.x, and Apache Pig itself.
- But most developers run higher versions of JDK on their development machines
through *SDKMAN!* and {*}Docker CE{*}, and JDK 7 is EOL. If Apache Pig can
provide a Docker Image, it will obviously help with quick testing.
> Providing a Docker Image for Apache Pig
> ---
>
> Key: PIG-5467
> URL: https://issues.apache.org/jira/browse/PIG-5467
> Project: Pig
> Issue Type: Improvement
>Reporter: Qiheng He
>Priority: Major
>
> - I noticed that [https://pig.apache.org/docs/r0.17.0/start.html] requires
> users to manually download JDK 7, Hadoop 2.x, and Apache Pig itself.
> - But most developers run higher versions of JDK on their development
> machines through *SDKMAN!* and {*}Docker CE{*}, and JDK 7 is EOL. If Apache
> Pig can provide a Docker Image, it will obviously help with quick testing.
> - This would help write unit tests on third-party libraries like
> *testcontainers-java* .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (PIG-5467) Providing a Docker Image for Apache Pig
Qiheng He created PIG-5467:
--
Summary: Providing a Docker Image for Apache Pig
Key: PIG-5467
URL: https://issues.apache.org/jira/browse/PIG-5467
Project: Pig
Issue Type: Improvement
Reporter: Qiheng He
- I noticed that https://pig.apache.org/docs/r0.17.0/start.html requires users
to manually download JDK 7, Hadoop 2.x, and Apache Pig itself.
- But most developers run higher versions of JDK on their development machines
through *SDKMAN!* and {*}Docker CE{*}, and JDK 7 is EOL. If Apache Pig can
provide a Docker Image, it will obviously help with quick testing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (PIG-5466) Enhance PigProgressNotificationListener interface to support extracting lineage
[ https://issues.apache.org/jira/browse/PIG-5466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Tolar updated PIG-5466: --- Release Note: * New methods added to PigProgressNotificationListener * Multiple notification listeners can be configured * PigRunner.run(args, null) is now ambiguous; replace with PigRunner.run(args, (PigProgressNotificationListener) null) Status: Patch Available (was: Open) > Enhance PigProgressNotificationListener interface to support extracting > lineage > --- > > Key: PIG-5466 > URL: https://issues.apache.org/jira/browse/PIG-5466 > Project: Pig > Issue Type: Improvement >Reporter: Rohini Palaniswamy >Assignee: Jeffrey Tolar >Priority: Major > Fix For: 0.18.0 > > Attachments: PIG-5466-v1.patch > > > Need changes to support multiple listeners and publish the script and the > Logical plan. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5466) Enhance PigProgressNotificationListener interface to support extracting lineage
[ https://issues.apache.org/jira/browse/PIG-5466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Tolar updated PIG-5466: --- Attachment: PIG-5466-v1.patch > Enhance PigProgressNotificationListener interface to support extracting > lineage > --- > > Key: PIG-5466 > URL: https://issues.apache.org/jira/browse/PIG-5466 > Project: Pig > Issue Type: Improvement >Reporter: Rohini Palaniswamy >Assignee: Jeffrey Tolar >Priority: Major > Fix For: 0.18.0 > > Attachments: PIG-5466-v1.patch > > > Need changes to support multiple listeners and publish the script and the > Logical plan. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5466) Enhance PigProgressNotificationListener interface to support extracting lineage
[ https://issues.apache.org/jira/browse/PIG-5466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated PIG-5466: Summary: Enhance PigProgressNotificationListener interface to support extracting lineage (was: Make changes to PigProgressNotificationListener interface to support extracting lineage) > Enhance PigProgressNotificationListener interface to support extracting > lineage > --- > > Key: PIG-5466 > URL: https://issues.apache.org/jira/browse/PIG-5466 > Project: Pig > Issue Type: Improvement >Reporter: Rohini Palaniswamy >Assignee: Jeffrey Tolar >Priority: Major > Fix For: 0.18.0 > > > Need changes to support multiple listeners and publish the script and the > Logical plan. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (PIG-5466) Make changes to PigProgressNotificationListener interface to support extracting lineage
Rohini Palaniswamy created PIG-5466: --- Summary: Make changes to PigProgressNotificationListener interface to support extracting lineage Key: PIG-5466 URL: https://issues.apache.org/jira/browse/PIG-5466 Project: Pig Issue Type: Improvement Reporter: Rohini Palaniswamy Assignee: Jeffrey Tolar Fix For: 0.18.0 Need changes to support multiple listeners and publish the script and the Logical plan. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (PIG-5465) Owasp filter out false positives
[ https://issues.apache.org/jira/browse/PIG-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi resolved PIG-5465. --- Fix Version/s: 0.19.0 Resolution: Fixed Thanks for the review Rohini! Committed to trunk. > Owasp filter out false positives > > > Key: PIG-5465 > URL: https://issues.apache.org/jira/browse/PIG-5465 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Minor > Fix For: 0.19.0 > > Attachments: pig-owasp.patch > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (PIG-5465) Owasp filter out false positives
[ https://issues.apache.org/jira/browse/PIG-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi reassigned PIG-5465: - Assignee: Koji Noguchi > Owasp filter out false positives > > > Key: PIG-5465 > URL: https://issues.apache.org/jira/browse/PIG-5465 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Minor > Attachments: pig-owasp.patch > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (PIG-5458) Update metrics-core.version
[ https://issues.apache.org/jira/browse/PIG-5458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi resolved PIG-5458. --- Fix Version/s: 0.19.0 Hadoop Flags: Reviewed Resolution: Fixed Thanks for the review Rohini! Committed to trunk. > Update metrics-core.version > > > Key: PIG-5458 > URL: https://issues.apache.org/jira/browse/PIG-5458 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Fix For: 0.19.0 > > Attachments: pig-5458-v01.patch > > > Hadoop3 uses metrics-core.version of 3.2.4 from io.dropwizard.metrics > and > Hadoop2 uses metrics-core.version of 3.0.1 from com.codahale.metrics. > I believe one from com.yammer.metrics (2.1.2) can be dropped. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (PIG-5451) Pig-on-Spark3 E2E Orc_Pushdown_5 failing
[ https://issues.apache.org/jira/browse/PIG-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi resolved PIG-5451. --- Fix Version/s: 0.19.0 Hadoop Flags: Reviewed Resolution: Fixed Thanks for the review Rohini! Committed to trunk. > Pig-on-Spark3 E2E Orc_Pushdown_5 failing > - > > Key: PIG-5451 > URL: https://issues.apache.org/jira/browse/PIG-5451 > Project: Pig > Issue Type: Bug >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Minor > Fix For: 0.19.0 > > Attachments: pig-9-5451-v01.patch > > > Test failing with > "java.lang.IllegalAccessError: class org.threeten.extra.chrono.HybridDate > cannot access its superclass org.threeten.extra.chrono.AbstractDate" -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (PIG-5420) Update accumulo dependency to 1.10.4
[ https://issues.apache.org/jira/browse/PIG-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi resolved PIG-5420. --- Fix Version/s: 0.19.0 (was: 0.18.1) Hadoop Flags: Reviewed Resolution: Fixed Thanks for the review Rohini! Committed to trunk. > Update accumulo dependency to 1.10.4 > > > Key: PIG-5420 > URL: https://issues.apache.org/jira/browse/PIG-5420 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Fix For: 0.19.0 > > Attachments: pig-5420-v01.patch, pig-9-5420-v02.patch > > > Following owasp/cve report. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5420) Update accumulo dependency to 1.10.4
[ https://issues.apache.org/jira/browse/PIG-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5420: -- Summary: Update accumulo dependency to 1.10.4 (was: Update accumulo dependency to 1.10.1) > Update accumulo dependency to 1.10.4 > > > Key: PIG-5420 > URL: https://issues.apache.org/jira/browse/PIG-5420 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Fix For: 0.18.1 > > Attachments: pig-5420-v01.patch, pig-9-5420-v02.patch > > > Following owasp/cve report. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (PIG-5463) Pig on Tez TestDateTime.testLocalExecution failing on hadoop3/tez-0.10
[
https://issues.apache.org/jira/browse/PIG-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi resolved PIG-5463.
---
Hadoop Flags: Reviewed
Resolution: Fixed
Thanks for the review Rohini!
Committed to trunk.
> Pig on Tez TestDateTime.testLocalExecution failing on hadoop3/tez-0.10
> --
>
> Key: PIG-5463
> URL: https://issues.apache.org/jira/browse/PIG-5463
> Project: Pig
> Issue Type: Test
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Minor
> Fix For: 0.19.0
>
> Attachments: pig-5463-v01.patch, pig-5463-v02.patch
>
>
> Somehow TestDateTime testLocalExecution started failing on Pig on Tez with
> hadoop3.
> {noformat}
> 2024-09-11 10:50:29,815 [IPC Server handler 30 on default port 34089] WARN
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor - Invalid
> resource ask by application appattempt_1726051802536_0001_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request! Cannot allocate containers as requested resource is less
> than 0! Requested resource type=[memory-mb], Requested resource= vCores:1>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.throwInvalidResourceException(SchedulerUtils.java:525)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.checkResourceRequestAgainstAvailableResource(SchedulerUtils.java:415)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:349)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:304)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:268)
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:254)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:93)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:434)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:105)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:621)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:589)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048)
> {noformat}
> Weird part is, it passes when tested alone or tested twice (with copy&paste).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (PIG-5463) Pig on Tez TestDateTime.testLocalExecution failing on hadoop3/tez-0.10
[
https://issues.apache.org/jira/browse/PIG-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17888440#comment-17888440
]
Rohini Palaniswamy commented on PIG-5463:
-
+1
> Pig on Tez TestDateTime.testLocalExecution failing on hadoop3/tez-0.10
> --
>
> Key: PIG-5463
> URL: https://issues.apache.org/jira/browse/PIG-5463
> Project: Pig
> Issue Type: Test
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Minor
> Fix For: 0.19.0
>
> Attachments: pig-5463-v01.patch, pig-5463-v02.patch
>
>
> Somehow TestDateTime testLocalExecution started failing on Pig on Tez with
> hadoop3.
> {noformat}
> 2024-09-11 10:50:29,815 [IPC Server handler 30 on default port 34089] WARN
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor - Invalid
> resource ask by application appattempt_1726051802536_0001_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request! Cannot allocate containers as requested resource is less
> than 0! Requested resource type=[memory-mb], Requested resource= vCores:1>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.throwInvalidResourceException(SchedulerUtils.java:525)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.checkResourceRequestAgainstAvailableResource(SchedulerUtils.java:415)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:349)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:304)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:268)
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:254)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:93)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:434)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:105)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:621)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:589)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048)
> {noformat}
> Weird part is, it passes when tested alone or tested twice (with copy&paste).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (PIG-5463) Pig on Tez TestDateTime.testLocalExecution failing on hadoop3/tez-0.10
[
https://issues.apache.org/jira/browse/PIG-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-5463:
--
Attachment: pig-5463-v02.patch
> Can you just rename TestLocalDateTime.java to TestDateTimeLocal.java so that
> both files appear next to each other ?
>
Make sense. Updated a patch.
> Pig on Tez TestDateTime.testLocalExecution failing on hadoop3/tez-0.10
> --
>
> Key: PIG-5463
> URL: https://issues.apache.org/jira/browse/PIG-5463
> Project: Pig
> Issue Type: Test
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Minor
> Fix For: 0.19.0
>
> Attachments: pig-5463-v01.patch, pig-5463-v02.patch
>
>
> Somehow TestDateTime testLocalExecution started failing on Pig on Tez with
> hadoop3.
> {noformat}
> 2024-09-11 10:50:29,815 [IPC Server handler 30 on default port 34089] WARN
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor - Invalid
> resource ask by application appattempt_1726051802536_0001_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request! Cannot allocate containers as requested resource is less
> than 0! Requested resource type=[memory-mb], Requested resource= vCores:1>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.throwInvalidResourceException(SchedulerUtils.java:525)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.checkResourceRequestAgainstAvailableResource(SchedulerUtils.java:415)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:349)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:304)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:268)
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:254)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:93)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:434)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:105)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:621)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:589)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048)
> {noformat}
> Weird part is, it passes when tested alone or tested twice (with copy&paste).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (PIG-5462) Always update Owasp version to latest
[ https://issues.apache.org/jira/browse/PIG-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi resolved PIG-5462. --- Fix Version/s: 0.19.0 Resolution: Fixed Thanks for the review Rohini! Committed to trunk. > Always update Owasp version to latest > -- > > Key: PIG-5462 > URL: https://issues.apache.org/jira/browse/PIG-5462 > Project: Pig > Issue Type: Test >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Fix For: 0.19.0 > > Attachments: pig-5462-v01.patch, pig-5462-v02.patch > > > While looking at owasp report, a lot of them were completely off. > (Like hadoop-shims-0.10.3 being reported as vulnerable.) > Using latest org.owasp/dependency-check-ant > (https://mvnrepository.com/artifact/org.owasp/dependency-check-ant) > seems to help cut down the false positives. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (PIG-5461) E2E environment variables ignored
[ https://issues.apache.org/jira/browse/PIG-5461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi resolved PIG-5461. --- Fix Version/s: 0.19.0 Resolution: Fixed Thanks for the review Rohini! Committed to trunk. > E2E environment variables ignored > - > > Key: PIG-5461 > URL: https://issues.apache.org/jira/browse/PIG-5461 > Project: Pig > Issue Type: Test >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Fix For: 0.19.0 > > Attachments: pig-5461-v01.patch > > > When running e2e against Hadoop3 and using hadoop2+oldpig for verification, I > was confused why environment variables like OLD_HADOOP_HOME were ignored. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (PIG-5460) Allow Tez to be launched from mapreduce job
[ https://issues.apache.org/jira/browse/PIG-5460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi resolved PIG-5460. --- Fix Version/s: 0.19.0 Resolution: Fixed Thanks for the review Rohini! Committed to trunk. > Allow Tez to be launched from mapreduce job > --- > > Key: PIG-5460 > URL: https://issues.apache.org/jira/browse/PIG-5460 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Minor > Fix For: 0.19.0 > > Attachments: pig-5460-v01.patch, pig-5460-v02.patch > > > It's like Oozie but not using Oozie launcher. > I would like to be able to submit Pig on Tez job from the mapper task. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (PIG-5454) Make ParallelGC the default Garbage Collection
[ https://issues.apache.org/jira/browse/PIG-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi resolved PIG-5454. --- Fix Version/s: 0.19.0 Hadoop Flags: Reviewed Resolution: Fixed Thanks for the review Rohini! Committed to trunk. > Make ParallelGC the default Garbage Collection > -- > > Key: PIG-5454 > URL: https://issues.apache.org/jira/browse/PIG-5454 > Project: Pig > Issue Type: Bug > Components: impl >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Fix For: 0.19.0 > > Attachments: pig-5454-v01.patch, pig-5454-v02.patch, > pig-5454-v03.patch, pig-5454-v04.patch > > > From JDK9 and beyond, G1GC became the default GC. > I've seen our users hitting OOM after migrating to recent jdk and the issue > going away after reverting back to ParallelGC. > Maybe the GC behavior assumed by SelfSpillBag does not work with G1GC. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (PIG-5449) TestEmptyInputDir failing on pig-on-spark3
[
https://issues.apache.org/jira/browse/PIG-5449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi resolved PIG-5449.
---
Fix Version/s: 0.19.0
Hadoop Flags: Reviewed
Resolution: Fixed
Thanks for the review Rohini!
Committed to trunk.
> TestEmptyInputDir failing on pig-on-spark3
> --
>
> Key: PIG-5449
> URL: https://issues.apache.org/jira/browse/PIG-5449
> Project: Pig
> Issue Type: Bug
> Components: spark
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Major
> Fix For: 0.19.0
>
> Attachments: pig-5449-v01.patch
>
>
> TestEmptyInputDir failing on pig-on-spark3 with
> {noformat:title=TestEmptyInputDir.testMergeJoinFailure}
> junit.framework.AssertionFailedError
> at
> org.apache.pig.test.TestEmptyInputDir.testMergeJoin(TestEmptyInputDir.java:141)
> {noformat}
> {noformat:title=TestEmptyInputDir.testGroupByFailure}
> junit.framework.AssertionFailedError
> at
> org.apache.pig.test.TestEmptyInputDir.testGroupBy(TestEmptyInputDir.java:80)
> {noformat}
> {noformat:title=TestEmptyInputDir.testBloomJoinOuterFailure}
> junit.framework.AssertionFailedError
> at
> org.apache.pig.test.TestEmptyInputDir.testBloomJoinOuter(TestEmptyInputDir.java:297)
> {noformat}
> {noformat:title=TestEmptyInputDir.testFRJoinFailure}
> junit.framework.AssertionFailedError
> at
> org.apache.pig.test.TestEmptyInputDir.testFRJoin(TestEmptyInputDir.java:171)
> {noformat}
> {noformat:title=TestEmptyInputDir.testBloomJoinFailure}
> junit.framework.AssertionFailedError
> at
> org.apache.pig.test.TestEmptyInputDir.testBloomJoin(TestEmptyInputDir.java:267)
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (PIG-5454) Make ParallelGC the default Garbage Collection
[ https://issues.apache.org/jira/browse/PIG-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17888416#comment-17888416 ] Rohini Palaniswamy commented on PIG-5454: - +1 > Make ParallelGC the default Garbage Collection > -- > > Key: PIG-5454 > URL: https://issues.apache.org/jira/browse/PIG-5454 > Project: Pig > Issue Type: Bug > Components: impl >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Attachments: pig-5454-v01.patch, pig-5454-v02.patch, > pig-5454-v03.patch, pig-5454-v04.patch > > > From JDK9 and beyond, G1GC became the default GC. > I've seen our users hitting OOM after migrating to recent jdk and the issue > going away after reverting back to ParallelGC. > Maybe the GC behavior assumed by SelfSpillBag does not work with G1GC. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5454) Make ParallelGC the default Garbage Collection
[ https://issues.apache.org/jira/browse/PIG-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5454: -- Attachment: pig-5454-v04.patch Uploading a new patch with "params". > Make ParallelGC the default Garbage Collection > -- > > Key: PIG-5454 > URL: https://issues.apache.org/jira/browse/PIG-5454 > Project: Pig > Issue Type: Bug > Components: impl >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Attachments: pig-5454-v01.patch, pig-5454-v02.patch, > pig-5454-v03.patch, pig-5454-v04.patch > > > From JDK9 and beyond, G1GC became the default GC. > I've seen our users hitting OOM after migrating to recent jdk and the issue > going away after reverting back to ParallelGC. > Maybe the GC behavior assumed by SelfSpillBag does not work with G1GC. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PIG-5465) Owasp filter out false positives
[ https://issues.apache.org/jira/browse/PIG-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17888387#comment-17888387 ] Rohini Palaniswamy commented on PIG-5465: - +1 > Owasp filter out false positives > > > Key: PIG-5465 > URL: https://issues.apache.org/jira/browse/PIG-5465 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Priority: Minor > Attachments: pig-owasp.patch > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5410) Support Python 3 for streaming_python
[ https://issues.apache.org/jira/browse/PIG-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkatasubrahmanian Narayanan updated PIG-5410: --- Attachment: PIG-5410-v5.patch > Support Python 3 for streaming_python > - > > Key: PIG-5410 > URL: https://issues.apache.org/jira/browse/PIG-5410 > Project: Pig > Issue Type: New Feature >Reporter: Rohini Palaniswamy >Assignee: Venkatasubrahmanian Narayanan >Priority: Major > Fix For: 0.18.0 > > Attachments: PIG-5410-v2.patch, PIG-5410-v3.patch, PIG-5410-v4.patch, > PIG-5410-v5.patch, PIG-5410.patch, pig-5410-v02.patch > > > Python 3 is incompatible with Python 2. We need to make it work with both. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (PIG-5465) Owasp filter out false positives
Koji Noguchi created PIG-5465: - Summary: Owasp filter out false positives Key: PIG-5465 URL: https://issues.apache.org/jira/browse/PIG-5465 Project: Pig Issue Type: Improvement Reporter: Koji Noguchi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5465) Owasp filter out false positives
[ https://issues.apache.org/jira/browse/PIG-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5465: -- Attachment: pig-owasp.patch > Owasp filter out false positives > > > Key: PIG-5465 > URL: https://issues.apache.org/jira/browse/PIG-5465 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Priority: Minor > Attachments: pig-owasp.patch > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PIG-5460) Allow Tez to be launched from mapreduce job
[ https://issues.apache.org/jira/browse/PIG-5460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17885424#comment-17885424 ] Rohini Palaniswamy commented on PIG-5460: - +1 > Allow Tez to be launched from mapreduce job > --- > > Key: PIG-5460 > URL: https://issues.apache.org/jira/browse/PIG-5460 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Minor > Attachments: pig-5460-v01.patch, pig-5460-v02.patch > > > It's like Oozie but not using Oozie launcher. > I would like to be able to submit Pig on Tez job from the mapper task. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5410) Support Python 3 for streaming_python
[ https://issues.apache.org/jira/browse/PIG-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkatasubrahmanian Narayanan updated PIG-5410: --- Attachment: PIG-5410-v4.patch > Support Python 3 for streaming_python > - > > Key: PIG-5410 > URL: https://issues.apache.org/jira/browse/PIG-5410 > Project: Pig > Issue Type: New Feature >Reporter: Rohini Palaniswamy >Assignee: Venkatasubrahmanian Narayanan >Priority: Major > Fix For: 0.18.0 > > Attachments: PIG-5410-v2.patch, PIG-5410-v3.patch, PIG-5410-v4.patch, > PIG-5410.patch, pig-5410-v02.patch > > > Python 3 is incompatible with Python 2. We need to make it work with both. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5410) Support Python 3 for streaming_python
[ https://issues.apache.org/jira/browse/PIG-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkatasubrahmanian Narayanan updated PIG-5410: --- Attachment: PIG-5410-v3.patch > Support Python 3 for streaming_python > - > > Key: PIG-5410 > URL: https://issues.apache.org/jira/browse/PIG-5410 > Project: Pig > Issue Type: New Feature >Reporter: Rohini Palaniswamy >Assignee: Venkatasubrahmanian Narayanan >Priority: Major > Fix For: 0.18.0 > > Attachments: PIG-5410-v2.patch, PIG-5410-v3.patch, PIG-5410.patch, > pig-5410-v02.patch > > > Python 3 is incompatible with Python 2. We need to make it work with both. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5460) Allow Tez to be launched from mapreduce job
[ https://issues.apache.org/jira/browse/PIG-5460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5460: -- Attachment: pig-5460-v02.patch Attaching your suggestions. pig-5460-v02.patch > Allow Tez to be launched from mapreduce job > --- > > Key: PIG-5460 > URL: https://issues.apache.org/jira/browse/PIG-5460 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Minor > Attachments: pig-5460-v01.patch, pig-5460-v02.patch > > > It's like Oozie but not using Oozie launcher. > I would like to be able to submit Pig on Tez job from the mapper task. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (PIG-5459) Jython_Checkin_3 e2e failing with NoClassDefFoundError (hadoop3)
[
https://issues.apache.org/jira/browse/PIG-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi resolved PIG-5459.
---
Fix Version/s: 0.19.0
Hadoop Flags: Reviewed
Resolution: Fixed
Thanks for the review Rohini!
Committed to trunk.
> Jython_Checkin_3 e2e failing with NoClassDefFoundError (hadoop3)
>
>
> Key: PIG-5459
> URL: https://issues.apache.org/jira/browse/PIG-5459
> Project: Pig
> Issue Type: Bug
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Minor
> Fix For: 0.19.0
>
> Attachments: pig-5459-v01.patch
>
>
> {noformat}
> turing_jython.conf/Jython_Checkin_3.pig", line 4, in _module_
> from org.apache.hadoop.conf import *
> java.lang.NoClassDefFoundError: Lorg/junit/rules/ExpectedException;
> at java.lang.Class.getDeclaredFields0(Native Method)
> at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
> at java.lang.Class.privateGetPublicFields(Class.java:2614)
> at java.lang.Class.getFields(Class.java:1557)
> at org.python.core.PyJavaType.init(PyJavaType.java:419)
> at org.python.core.PyType.createType(PyType.java:1523)
> at org.python.core.PyType.addFromClass(PyType.java:1462)
> at org.python.core.PyType.fromClass(PyType.java:1551)
> at
> org.python.core.adapter.ClassicPyObjectAdapter$6.adapt(ClassicPyObjectAdapter.java:77)
> at
> org.python.core.adapter.ExtensiblePyObjectAdapter.adapt(ExtensiblePyObjectAdapter.java:44)
> at
> org.python.core.adapter.ClassicPyObjectAdapter.adapt(ClassicPyObjectAdapter.java:131)
> at org.python.core.Py.java2py(Py.java:2017)
> at org.python.core.PyJavaPackage.addClass(PyJavaPackage.java:86)
> at
> org.python.core.packagecache.PackageManager.basicDoDir(PackageManager.java:113)
> at
> org.python.core.packagecache.SysPackageManager.doDir(SysPackageManager.java:148)
> at org.python.core.PyJavaPackage.fillDir(PyJavaPackage.java:120)
> at org.python.core.imp.importAll(imp.java:1189)
> at org.python.core.imp.importAll(imp.java:1177)
> at
> org.python.pycode._pyx0.f$0(/tmp/yarn-local/usercache/.../gtrain-1722336537-turing_jython.conf/Jython_Checkin_3.pig:8)
> at
> org.python.pycode._pyx0.call_function(/tmp/yarn-local/usercache...gtrain-1722336537-tu/ring_jython.conf/Jython_Checkin_3.pig)
> at org.python.core.PyTableCode.call(PyTableCode.java:171)
> at org.python.core.PyCode.call(PyCode.java:18)
> at org.python.core.Py.runCode(Py.java:1614)
> at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:296)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:217)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:440)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:424)
> at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:310)
> at org.apache.pig.Main.runEmbeddedScript(Main.java:1096)
> at org.apache.pig.Main.run(Main.java:584)
> at org.apache.pig.Main.main(Main.java:175)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:328)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:241)
> Caused by: java.lang.ClassNotFoundException: org.junit.rules.ExpectedException
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 37 more
> java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError:
> Lorg/junit/rules/ExpectedException;
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (PIG-5457) Upgrade Zookeeper to 3.7.2 (from 3.5.7)
[ https://issues.apache.org/jira/browse/PIG-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi resolved PIG-5457. --- Hadoop Flags: Reviewed Resolution: Fixed Thanks for the review Rohini! Committed to trunk. > Upgrade Zookeeper to 3.7.2 (from 3.5.7) > --- > > Key: PIG-5457 > URL: https://issues.apache.org/jira/browse/PIG-5457 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Fix For: 0.19.0 > > Attachments: pig-5457-v01.patch, pig-5457-v02.patch > > > As mentioned in PIG-5456, zookeeper-3.5.7 dependency pulls in > log4j-1.2.17.jar that we want to avoid. Updating to 3.6.4, making it same as > the dependency from hadoop 3.3.6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5456) Upgrade Spark to 3.4.3
[
https://issues.apache.org/jira/browse/PIG-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-5456:
--
Attachment: pig-5456-v03.patch
Due to how I created the patch, path was pointing to /tmp.
{code}
--- /tmp/libraries.properties 2024-06-21 12:01:27
+++ ivy/libraries.properties2024-06-21 12:01:50
{code}
Patch still applied (probably due to second path being correct) but uploading a
new patch with correct path just in case.
> Upgrade Spark to 3.4.3
> --
>
> Key: PIG-5456
> URL: https://issues.apache.org/jira/browse/PIG-5456
> Project: Pig
> Issue Type: Improvement
> Components: spark
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Major
> Fix For: 0.19.0
>
> Attachments: pig-5456-v01.patch, pig-5456-v02.patch,
> pig-5456-v03.patch
>
>
> Major blocker for upgrading to Spark 3.4.3 was Spark started using log4j2.
> Simple upgrade failing a lot of tests with
> {noformat}
> java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter
> overrides final method getTimeStamp.()J {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (PIG-5456) Upgrade Spark to 3.4.3
[
https://issues.apache.org/jira/browse/PIG-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi resolved PIG-5456.
---
Hadoop Flags: Reviewed
Resolution: Fixed
Thanks for the review Rohini!
Committed to trunk (0.19) for now but we will most likely re-branch 0.18 from
latest trunk.
> Upgrade Spark to 3.4.3
> --
>
> Key: PIG-5456
> URL: https://issues.apache.org/jira/browse/PIG-5456
> Project: Pig
> Issue Type: Improvement
> Components: spark
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Major
> Fix For: 0.19.0
>
> Attachments: pig-5456-v01.patch, pig-5456-v02.patch
>
>
> Major blocker for upgrading to Spark 3.4.3 was Spark started using log4j2.
> Simple upgrade failing a lot of tests with
> {noformat}
> java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter
> overrides final method getTimeStamp.()J {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (PIG-5419) Upgrade Joda time version
[ https://issues.apache.org/jira/browse/PIG-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated PIG-5419: Fix Version/s: 0.18.0 (was: 0.18.1) Hadoop Flags: Reviewed Patch Info: Patch Available +1. Thanks Venkat > Upgrade Joda time version > - > > Key: PIG-5419 > URL: https://issues.apache.org/jira/browse/PIG-5419 > Project: Pig > Issue Type: Improvement >Reporter: Venkatasubrahmanian Narayanan >Assignee: Venkatasubrahmanian Narayanan >Priority: Minor > Fix For: 0.18.0 > > Attachments: PIG-5419-v2.patch, PIG-5419.patch > > > Pig depends on an older version of Joda time, which can result in conflicts > with other versions in some workflows. Upgrading it to the latest > version(2.10.13) will resolve Pig's side of such issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5410) Support Python 3 for streaming_python
[ https://issues.apache.org/jira/browse/PIG-5410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkatasubrahmanian Narayanan updated PIG-5410: --- Attachment: PIG-5410-v2.patch > Support Python 3 for streaming_python > - > > Key: PIG-5410 > URL: https://issues.apache.org/jira/browse/PIG-5410 > Project: Pig > Issue Type: New Feature >Reporter: Rohini Palaniswamy >Assignee: Venkatasubrahmanian Narayanan >Priority: Major > Fix For: 0.18.0 > > Attachments: PIG-5410-v2.patch, PIG-5410.patch, pig-5410-v02.patch > > > Python 3 is incompatible with Python 2. We need to make it work with both. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5445) TestTezCompiler.testMergeCogroup fails whenever config is updated
[
https://issues.apache.org/jira/browse/PIG-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-5445:
--
Status: Patch Available (was: Open)
> TestTezCompiler.testMergeCogroup fails whenever config is updated
> -
>
> Key: PIG-5445
> URL: https://issues.apache.org/jira/browse/PIG-5445
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.19.0
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Minor
> Attachments: pig-5445-v01.patch
>
>
> TestTezCompiler.testMergeCogroup started failing after upgrading Tez (and
> config that comes with it).
> {noformat}
> testMergeCogroupFailure
> expected:
> <|---a:
> Load(file:///tmp/input1:org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MergeJoinIndexer('org.a
>
> pache.pig.test.TestMapSideCogroup$DummyCollectableLoader','.../doPMfwFKyneZ','eNq9[fWtsHFeWXvEhWm9Ls...XOuwcT+fzW1+yM]=','a_1-0','scope','...>
>
> but was:
> <|---a:
> Load(file:///tmp/input1:org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MergeJoinIndexer('org.a
>
> pache.pig.test.TestMapSideCogroup$DummyCollectableLoader','.../doPMfwFKyneZ','eNq9[V01sG0UUnmycxHWSN...DyC6P4Drk9M9w=]=','a_1-0','scope','...>
> at org.apache.pig.tez.TestTezCompiler.run(TestTezCompiler.java:1472)
> at
> org.apache.pig.tez.TestTezCompiler.testMergeCogroup(TestTezCompiler.java:292)
> {noformat}
> (edited the diff above a bit to make it easier to identify where the
> difference was)
> Basically 3rd argument to MergeJoinIndexer differed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (PIG-5444) TestFRJoin.testFRJoinOut7 and testFRJoinOut8 failing with Edge already defined error on Tez
[
https://issues.apache.org/jira/browse/PIG-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-5444:
--
Status: Patch Available (was: Open)
> TestFRJoin.testFRJoinOut7 and testFRJoinOut8 failing with Edge already
> defined error on Tez
> ---
>
> Key: PIG-5444
> URL: https://issues.apache.org/jira/browse/PIG-5444
> Project: Pig
> Issue Type: Bug
> Components: tez
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Major
> Attachments: pig-5444-v02.patch, pig-5444-v03.patch
>
>
> With Tez, when testing individual tests (TestFRJoin.testFRJoinOut7 and
> testFRJoinOut8) separately, they pass the tests. But when entire TestFRJoin
> is run, these two tests on Tez are failing with
> {noformat}
> Unable to open iterator for alias E
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
> open iterator for alias E
> at org.apache.pig.PigServer.openIterator(PigServer.java:1024)
> at org.apache.pig.test.TestFRJoin.testFRJoinOut7(TestFRJoin.java:409)
> Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias E
> at org.apache.pig.PigServer.storeEx(PigServer.java:1127)
> at org.apache.pig.PigServer.store(PigServer.java:1086)
> at org.apache.pig.PigServer.openIterator(PigServer.java:999)
> Caused by:
> org.apache.pig.backend.hadoop.executionengine.JobCreationException: ERROR
> 2017: Internal error creating job configuration.
> at
> org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:153)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.compile(TezJobCompiler.java:81)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:200)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1479)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464)
> at org.apache.pig.PigServer.storeEx(PigServer.java:1123)
> Caused by: java.lang.IllegalArgumentException: Edge [scope-632 :
> org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor] ->
> [scope-628 :
> org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor] ({
> BROADCAST : org.apache.tez.runtime.library.input.UnorderedKVInput >>
> PERSISTED >> org.apache.tez.runtime.library.output.UnorderedKVOutput >>
> NullEdgeManager }) already defined!
> at org.apache.tez.dag.api.DAG.addEdge(DAG.java:296)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder.visitTezOp(TezDagBuilder.java:410)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:265)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:56)
> at
> org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:87)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.buildDAG(TezJobCompiler.java:69)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:120)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (PIG-5419) Upgrade Joda time version
[ https://issues.apache.org/jira/browse/PIG-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884748#comment-17884748 ] Venkatasubrahmanian Narayanan commented on PIG-5419: Updated to 2.13.0. > Upgrade Joda time version > - > > Key: PIG-5419 > URL: https://issues.apache.org/jira/browse/PIG-5419 > Project: Pig > Issue Type: Improvement >Reporter: Venkatasubrahmanian Narayanan >Assignee: Venkatasubrahmanian Narayanan >Priority: Minor > Fix For: 0.18.1 > > Attachments: PIG-5419-v2.patch, PIG-5419.patch > > > Pig depends on an older version of Joda time, which can result in conflicts > with other versions in some workflows. Upgrading it to the latest > version(2.10.13) will resolve Pig's side of such issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5419) Upgrade Joda time version
[ https://issues.apache.org/jira/browse/PIG-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkatasubrahmanian Narayanan updated PIG-5419: --- Attachment: PIG-5419-v2.patch > Upgrade Joda time version > - > > Key: PIG-5419 > URL: https://issues.apache.org/jira/browse/PIG-5419 > Project: Pig > Issue Type: Improvement >Reporter: Venkatasubrahmanian Narayanan >Assignee: Venkatasubrahmanian Narayanan >Priority: Minor > Fix For: 0.18.1 > > Attachments: PIG-5419-v2.patch, PIG-5419.patch > > > Pig depends on an older version of Joda time, which can result in conflicts > with other versions in some workflows. Upgrading it to the latest > version(2.10.13) will resolve Pig's side of such issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5419) Upgrade Joda time version
[ https://issues.apache.org/jira/browse/PIG-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkatasubrahmanian Narayanan updated PIG-5419: --- Attachment: (was: PIG-5419.patch-v2.txt) > Upgrade Joda time version > - > > Key: PIG-5419 > URL: https://issues.apache.org/jira/browse/PIG-5419 > Project: Pig > Issue Type: Improvement >Reporter: Venkatasubrahmanian Narayanan >Assignee: Venkatasubrahmanian Narayanan >Priority: Minor > Fix For: 0.18.1 > > Attachments: PIG-5419.patch > > > Pig depends on an older version of Joda time, which can result in conflicts > with other versions in some workflows. Upgrading it to the latest > version(2.10.13) will resolve Pig's side of such issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5419) Upgrade Joda time version
[ https://issues.apache.org/jira/browse/PIG-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkatasubrahmanian Narayanan updated PIG-5419: --- Attachment: PIG-5419.patch-v2.txt > Upgrade Joda time version > - > > Key: PIG-5419 > URL: https://issues.apache.org/jira/browse/PIG-5419 > Project: Pig > Issue Type: Improvement >Reporter: Venkatasubrahmanian Narayanan >Assignee: Venkatasubrahmanian Narayanan >Priority: Minor > Fix For: 0.18.1 > > Attachments: PIG-5419.patch, PIG-5419.patch-v2.txt > > > Pig depends on an older version of Joda time, which can result in conflicts > with other versions in some workflows. Upgrading it to the latest > version(2.10.13) will resolve Pig's side of such issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5380) SortedDataBag hitting ConcurrentModificationException or producing incorrect output in a corner-case
[
https://issues.apache.org/jira/browse/PIG-5380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-5380:
--
Status: Open (was: Patch Available)
We haven't seen this issue pop up for a long time. We will revisit once 0.18
is released.
> SortedDataBag hitting ConcurrentModificationException or producing incorrect
> output in a corner-case
> -
>
> Key: PIG-5380
> URL: https://issues.apache.org/jira/browse/PIG-5380
> Project: Pig
> Issue Type: Bug
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Major
> Attachments: pig-5380-v01.patch, pig-5380-v02.patch,
> pig-5380-v03.patch
>
>
> User had a UDF that created large SortedDataBag. This UDF was failing with
> {noformat}
> java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
> at java.util.ArrayList$Itr.next(ArrayList.java:851)
> at
> org.apache.pig.data.SortedDataBag$SortedDataBagIterator.readFromPriorityQ(SortedDataBag.java:346)
> at
> org.apache.pig.data.SortedDataBag$SortedDataBagIterator.next(SortedDataBag.java:322)
> at
> org.apache.pig.data.SortedDataBag$SortedDataBagIterator.hasNext(SortedDataBag.java:235)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (PIG-5380) SortedDataBag hitting ConcurrentModificationException or producing incorrect output in a corner-case
[
https://issues.apache.org/jira/browse/PIG-5380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884310#comment-17884310
]
Rohini Palaniswamy commented on PIG-5380:
-
I think moving the reading from memory before the spill files might have
problems with the ordering.
> SortedDataBag hitting ConcurrentModificationException or producing incorrect
> output in a corner-case
> -
>
> Key: PIG-5380
> URL: https://issues.apache.org/jira/browse/PIG-5380
> Project: Pig
> Issue Type: Bug
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Major
> Attachments: pig-5380-v01.patch, pig-5380-v02.patch,
> pig-5380-v03.patch
>
>
> User had a UDF that created large SortedDataBag. This UDF was failing with
> {noformat}
> java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
> at java.util.ArrayList$Itr.next(ArrayList.java:851)
> at
> org.apache.pig.data.SortedDataBag$SortedDataBagIterator.readFromPriorityQ(SortedDataBag.java:346)
> at
> org.apache.pig.data.SortedDataBag$SortedDataBagIterator.next(SortedDataBag.java:322)
> at
> org.apache.pig.data.SortedDataBag$SortedDataBagIterator.hasNext(SortedDataBag.java:235)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (PIG-5454) Make ParallelGC the default Garbage Collection
[ https://issues.apache.org/jira/browse/PIG-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884287#comment-17884287 ] Rohini Palaniswamy commented on PIG-5454: - Just one minor comment. Make it params instead of param. i.e public static final String PIG_GC_DEFAULT_PARAMS = "pig.gc.default.params"; > Make ParallelGC the default Garbage Collection > -- > > Key: PIG-5454 > URL: https://issues.apache.org/jira/browse/PIG-5454 > Project: Pig > Issue Type: Bug > Components: impl >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Attachments: pig-5454-v01.patch, pig-5454-v02.patch, > pig-5454-v03.patch > > > From JDK9 and beyond, G1GC became the default GC. > I've seen our users hitting OOM after migrating to recent jdk and the issue > going away after reverting back to ParallelGC. > Maybe the GC behavior assumed by SelfSpillBag does not work with G1GC. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PIG-5456) Upgrade Spark to 3.4.3
[
https://issues.apache.org/jira/browse/PIG-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884286#comment-17884286
]
Rohini Palaniswamy commented on PIG-5456:
-
+1
> Upgrade Spark to 3.4.3
> --
>
> Key: PIG-5456
> URL: https://issues.apache.org/jira/browse/PIG-5456
> Project: Pig
> Issue Type: Improvement
> Components: spark
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Major
> Fix For: 0.19.0
>
> Attachments: pig-5456-v01.patch, pig-5456-v02.patch
>
>
> Major blocker for upgrading to Spark 3.4.3 was Spark started using log4j2.
> Simple upgrade failing a lot of tests with
> {noformat}
> java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter
> overrides final method getTimeStamp.()J {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (PIG-5464) Move off from jackson-mapper-asl and jackson-core-asl
[ https://issues.apache.org/jira/browse/PIG-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi reassigned PIG-5464: - Attachment: pig-5464-jackson_avro.patch Assignee: Koji Noguchi This patch is not to me committed. Only works for hadoop3 version. If we were to commit, we probably need a shim approach. > Move off from jackson-mapper-asl and jackson-core-asl > - > > Key: PIG-5464 > URL: https://issues.apache.org/jira/browse/PIG-5464 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Attachments: pig-5464-jackson_avro.patch > > > Similar to HADOOP-15983 and SPARK-30466, we need to move off from > jackson-mapper-asl-1.9.13 and jackson-core-asl-1.9.13. > However, this is only possible for Hadoop3. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (PIG-5464) Move off from jackson-mapper-asl and jackson-core-asl
[ https://issues.apache.org/jira/browse/PIG-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883093#comment-17883093 ] Koji Noguchi edited comment on PIG-5464 at 9/19/24 6:45 PM: This patch is not to be committed. Only works for hadoop3 version. If we were to commit, we probably need a shim approach. was (Author: knoguchi): This patch is not to me committed. Only works for hadoop3 version. If we were to commit, we probably need a shim approach. > Move off from jackson-mapper-asl and jackson-core-asl > - > > Key: PIG-5464 > URL: https://issues.apache.org/jira/browse/PIG-5464 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Attachments: pig-5464-jackson_avro.patch > > > Similar to HADOOP-15983 and SPARK-30466, we need to move off from > jackson-mapper-asl-1.9.13 and jackson-core-asl-1.9.13. > However, this is only possible for Hadoop3. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (PIG-5464) Move off from jackson-mapper-asl and jackson-core-asl
Koji Noguchi created PIG-5464: - Summary: Move off from jackson-mapper-asl and jackson-core-asl Key: PIG-5464 URL: https://issues.apache.org/jira/browse/PIG-5464 Project: Pig Issue Type: Improvement Reporter: Koji Noguchi Similar to HADOOP-15983 and SPARK-30466, we need to move off from jackson-mapper-asl-1.9.13 and jackson-core-asl-1.9.13. However, this is only possible for Hadoop3. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PIG-5459) Jython_Checkin_3 e2e failing with NoClassDefFoundError (hadoop3)
[
https://issues.apache.org/jira/browse/PIG-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882811#comment-17882811
]
Rohini Palaniswamy commented on PIG-5459:
-
+1
> Jython_Checkin_3 e2e failing with NoClassDefFoundError (hadoop3)
>
>
> Key: PIG-5459
> URL: https://issues.apache.org/jira/browse/PIG-5459
> Project: Pig
> Issue Type: Bug
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Minor
> Attachments: pig-5459-v01.patch
>
>
> {noformat}
> turing_jython.conf/Jython_Checkin_3.pig", line 4, in _module_
> from org.apache.hadoop.conf import *
> java.lang.NoClassDefFoundError: Lorg/junit/rules/ExpectedException;
> at java.lang.Class.getDeclaredFields0(Native Method)
> at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
> at java.lang.Class.privateGetPublicFields(Class.java:2614)
> at java.lang.Class.getFields(Class.java:1557)
> at org.python.core.PyJavaType.init(PyJavaType.java:419)
> at org.python.core.PyType.createType(PyType.java:1523)
> at org.python.core.PyType.addFromClass(PyType.java:1462)
> at org.python.core.PyType.fromClass(PyType.java:1551)
> at
> org.python.core.adapter.ClassicPyObjectAdapter$6.adapt(ClassicPyObjectAdapter.java:77)
> at
> org.python.core.adapter.ExtensiblePyObjectAdapter.adapt(ExtensiblePyObjectAdapter.java:44)
> at
> org.python.core.adapter.ClassicPyObjectAdapter.adapt(ClassicPyObjectAdapter.java:131)
> at org.python.core.Py.java2py(Py.java:2017)
> at org.python.core.PyJavaPackage.addClass(PyJavaPackage.java:86)
> at
> org.python.core.packagecache.PackageManager.basicDoDir(PackageManager.java:113)
> at
> org.python.core.packagecache.SysPackageManager.doDir(SysPackageManager.java:148)
> at org.python.core.PyJavaPackage.fillDir(PyJavaPackage.java:120)
> at org.python.core.imp.importAll(imp.java:1189)
> at org.python.core.imp.importAll(imp.java:1177)
> at
> org.python.pycode._pyx0.f$0(/tmp/yarn-local/usercache/.../gtrain-1722336537-turing_jython.conf/Jython_Checkin_3.pig:8)
> at
> org.python.pycode._pyx0.call_function(/tmp/yarn-local/usercache...gtrain-1722336537-tu/ring_jython.conf/Jython_Checkin_3.pig)
> at org.python.core.PyTableCode.call(PyTableCode.java:171)
> at org.python.core.PyCode.call(PyCode.java:18)
> at org.python.core.Py.runCode(Py.java:1614)
> at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:296)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:217)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:440)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:424)
> at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:310)
> at org.apache.pig.Main.runEmbeddedScript(Main.java:1096)
> at org.apache.pig.Main.run(Main.java:584)
> at org.apache.pig.Main.main(Main.java:175)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:328)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:241)
> Caused by: java.lang.ClassNotFoundException: org.junit.rules.ExpectedException
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 37 more
> java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError:
> Lorg/junit/rules/ExpectedException;
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (PIG-5451) Pig-on-Spark3 E2E Orc_Pushdown_5 failing
[ https://issues.apache.org/jira/browse/PIG-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882810#comment-17882810 ] Rohini Palaniswamy commented on PIG-5451: - +1 > Pig-on-Spark3 E2E Orc_Pushdown_5 failing > - > > Key: PIG-5451 > URL: https://issues.apache.org/jira/browse/PIG-5451 > Project: Pig > Issue Type: Bug >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Minor > Attachments: pig-9-5451-v01.patch > > > Test failing with > "java.lang.IllegalAccessError: class org.threeten.extra.chrono.HybridDate > cannot access its superclass org.threeten.extra.chrono.AbstractDate" -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PIG-5420) Update accumulo dependency to 1.10.1
[ https://issues.apache.org/jira/browse/PIG-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882809#comment-17882809 ] Rohini Palaniswamy commented on PIG-5420: - +1 > Update accumulo dependency to 1.10.1 > > > Key: PIG-5420 > URL: https://issues.apache.org/jira/browse/PIG-5420 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Fix For: 0.18.1 > > Attachments: pig-5420-v01.patch, pig-9-5420-v02.patch > > > Following owasp/cve report. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5420) Update accumulo dependency to 1.10.1
[ https://issues.apache.org/jira/browse/PIG-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5420: -- Attachment: pig-9-5420-v02.patch > Update accumulo dependency to 1.10.1 > > > Key: PIG-5420 > URL: https://issues.apache.org/jira/browse/PIG-5420 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Fix For: 0.18.1 > > Attachments: pig-5420-v01.patch, pig-9-5420-v02.patch > > > Following owasp/cve report. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PIG-5420) Update accumulo dependency to 1.10.1
[ https://issues.apache.org/jira/browse/PIG-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882808#comment-17882808 ] Koji Noguchi commented on PIG-5420: --- Uploaded pig-9-5420-v02.patch > Update accumulo dependency to 1.10.1 > > > Key: PIG-5420 > URL: https://issues.apache.org/jira/browse/PIG-5420 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Fix For: 0.18.1 > > Attachments: pig-5420-v01.patch, pig-9-5420-v02.patch > > > Following owasp/cve report. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PIG-5460) Allow Tez to be launched from mapreduce job
[
https://issues.apache.org/jira/browse/PIG-5460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882806#comment-17882806
]
Rohini Palaniswamy commented on PIG-5460:
-
Change should just be
{code:java}
String tokenFile = System.getenv("HADOOP_TOKEN_FILE_LOCATION")
if(tokenFile != null && globalConf.get(MRConfiguration.JOB_CREDENTIALS_BINARY)
== null) {
globalConf.set(MRConfiguration.JOB_CREDENTIALS_BINARY, tokenFile);
globalConf.set("tez.credentials.path", tokenFile);
}
{code}
SecurityHelper.populateTokenCache will take care of reading from that. It would
be even better if you can put the above into a
configureCredentialFile(Configuration conf) method in SecurityHelper instead of
TezDAGBuilder and just call it from there, so that all related code is in one
place.
> Allow Tez to be launched from mapreduce job
> ---
>
> Key: PIG-5460
> URL: https://issues.apache.org/jira/browse/PIG-5460
> Project: Pig
> Issue Type: Improvement
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Minor
> Attachments: pig-5460-v01.patch
>
>
> It's like Oozie but not using Oozie launcher.
> I would like to be able to submit Pig on Tez job from the mapper task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (PIG-5458) Update metrics-core.version
[ https://issues.apache.org/jira/browse/PIG-5458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882807#comment-17882807 ] Rohini Palaniswamy commented on PIG-5458: - +1 > Update metrics-core.version > > > Key: PIG-5458 > URL: https://issues.apache.org/jira/browse/PIG-5458 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Attachments: pig-5458-v01.patch > > > Hadoop3 uses metrics-core.version of 3.2.4 from io.dropwizard.metrics > and > Hadoop2 uses metrics-core.version of 3.0.1 from com.codahale.metrics. > I believe one from com.yammer.metrics (2.1.2) can be dropped. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PIG-5461) E2E environment variables ignored
[ https://issues.apache.org/jira/browse/PIG-5461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882803#comment-17882803 ] Rohini Palaniswamy commented on PIG-5461: - +1 > E2E environment variables ignored > - > > Key: PIG-5461 > URL: https://issues.apache.org/jira/browse/PIG-5461 > Project: Pig > Issue Type: Test >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Attachments: pig-5461-v01.patch > > > When running e2e against Hadoop3 and using hadoop2+oldpig for verification, I > was confused why environment variables like OLD_HADOOP_HOME were ignored. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PIG-5462) Always update Owasp version to latest
[ https://issues.apache.org/jira/browse/PIG-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882802#comment-17882802 ] Rohini Palaniswamy commented on PIG-5462: - +1 > Always update Owasp version to latest > -- > > Key: PIG-5462 > URL: https://issues.apache.org/jira/browse/PIG-5462 > Project: Pig > Issue Type: Test >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Attachments: pig-5462-v01.patch, pig-5462-v02.patch > > > While looking at owasp report, a lot of them were completely off. > (Like hadoop-shims-0.10.3 being reported as vulnerable.) > Using latest org.owasp/dependency-check-ant > (https://mvnrepository.com/artifact/org.owasp/dependency-check-ant) > seems to help cut down the false positives. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PIG-5457) Upgrade Zookeeper to 3.7.2 (from 3.5.7)
[ https://issues.apache.org/jira/browse/PIG-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882801#comment-17882801 ] Rohini Palaniswamy commented on PIG-5457: - +1 > Upgrade Zookeeper to 3.7.2 (from 3.5.7) > --- > > Key: PIG-5457 > URL: https://issues.apache.org/jira/browse/PIG-5457 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Fix For: 0.19.0 > > Attachments: pig-5457-v01.patch, pig-5457-v02.patch > > > As mentioned in PIG-5456, zookeeper-3.5.7 dependency pulls in > log4j-1.2.17.jar that we want to avoid. Updating to 3.6.4, making it same as > the dependency from hadoop 3.3.6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PIG-5463) Pig on Tez TestDateTime.testLocalExecution failing on hadoop3/tez-0.10
[
https://issues.apache.org/jira/browse/PIG-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882798#comment-17882798
]
Rohini Palaniswamy commented on PIG-5463:
-
Can you just rename TestLocalDateTime.java to TestDateTimeLocal.java so that
both files appear next to each other ?
> Pig on Tez TestDateTime.testLocalExecution failing on hadoop3/tez-0.10
> --
>
> Key: PIG-5463
> URL: https://issues.apache.org/jira/browse/PIG-5463
> Project: Pig
> Issue Type: Test
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Minor
> Fix For: 0.19.0
>
> Attachments: pig-5463-v01.patch
>
>
> Somehow TestDateTime testLocalExecution started failing on Pig on Tez with
> hadoop3.
> {noformat}
> 2024-09-11 10:50:29,815 [IPC Server handler 30 on default port 34089] WARN
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor - Invalid
> resource ask by application appattempt_1726051802536_0001_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request! Cannot allocate containers as requested resource is less
> than 0! Requested resource type=[memory-mb], Requested resource= vCores:1>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.throwInvalidResourceException(SchedulerUtils.java:525)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.checkResourceRequestAgainstAvailableResource(SchedulerUtils.java:415)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:349)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:304)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:268)
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:254)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:93)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:434)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:105)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:621)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:589)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048)
> {noformat}
> Weird part is, it passes when tested alone or tested twice (with copy&paste).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (PIG-5463) Pig on Tez TestDateTime.testLocalExecution failing on hadoop3/tez-0.10
[
https://issues.apache.org/jira/browse/PIG-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi reassigned PIG-5463:
-
Assignee: Koji Noguchi
> Pig on Tez TestDateTime.testLocalExecution failing on hadoop3/tez-0.10
> --
>
> Key: PIG-5463
> URL: https://issues.apache.org/jira/browse/PIG-5463
> Project: Pig
> Issue Type: Test
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Minor
> Fix For: 0.19.0
>
> Attachments: pig-5463-v01.patch
>
>
> Somehow TestDateTime testLocalExecution started failing on Pig on Tez with
> hadoop3.
> {noformat}
> 2024-09-11 10:50:29,815 [IPC Server handler 30 on default port 34089] WARN
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor - Invalid
> resource ask by application appattempt_1726051802536_0001_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request! Cannot allocate containers as requested resource is less
> than 0! Requested resource type=[memory-mb], Requested resource= vCores:1>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.throwInvalidResourceException(SchedulerUtils.java:525)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.checkResourceRequestAgainstAvailableResource(SchedulerUtils.java:415)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:349)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:304)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:268)
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:254)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:93)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:434)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:105)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:621)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:589)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048)
> {noformat}
> Weird part is, it passes when tested alone or tested twice (with copy&paste).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (PIG-5463) Pig on Tez TestDateTime.testLocalExecution failing on hadoop3/tez-0.10
[
https://issues.apache.org/jira/browse/PIG-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-5463:
--
Attachment: pig-5463-v01.patch
Fix Version/s: 0.19.0
I believe this has something to do with having both
{code}
pigServer = new PigServer(cluster.getExecType(),
cluster.getProperties());
pigServerLocal = new PigServer(Util.getLocalTestMode(), new
Properties());
{code}
Initialization of pigServer adds hdfs config etc.
For now, splitting the test file into two to stabilize the test.
Uploaded pig-5463-v01.patch.
> Pig on Tez TestDateTime.testLocalExecution failing on hadoop3/tez-0.10
> --
>
> Key: PIG-5463
> URL: https://issues.apache.org/jira/browse/PIG-5463
> Project: Pig
> Issue Type: Test
>Reporter: Koji Noguchi
>Priority: Minor
> Fix For: 0.19.0
>
> Attachments: pig-5463-v01.patch
>
>
> Somehow TestDateTime testLocalExecution started failing on Pig on Tez with
> hadoop3.
> {noformat}
> 2024-09-11 10:50:29,815 [IPC Server handler 30 on default port 34089] WARN
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor - Invalid
> resource ask by application appattempt_1726051802536_0001_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request! Cannot allocate containers as requested resource is less
> than 0! Requested resource type=[memory-mb], Requested resource= vCores:1>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.throwInvalidResourceException(SchedulerUtils.java:525)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.checkResourceRequestAgainstAvailableResource(SchedulerUtils.java:415)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:349)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:304)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:268)
> at
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:254)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:93)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:434)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:105)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:621)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:589)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048)
> {noformat}
> Weird part is, it passes when tested alone or tested twice (with copy&paste).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (PIG-5463) Pig on Tez TestDateTime.testLocalExecution failing on hadoop3/tez-0.10
Koji Noguchi created PIG-5463:
-
Summary: Pig on Tez TestDateTime.testLocalExecution failing on
hadoop3/tez-0.10
Key: PIG-5463
URL: https://issues.apache.org/jira/browse/PIG-5463
Project: Pig
Issue Type: Test
Reporter: Koji Noguchi
Somehow TestDateTimetestLocalExecution started failing on Pig on Tez with
hadoop3.
{noformat}
2024-09-11 10:50:29,815 [IPC Server handler 30 on default port 34089] WARN
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor - Invalid
resource ask by application appattempt_1726051802536_0001_01
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request! Cannot allocate containers as requested resource is less than
0! Requested resource type=[memory-mb], Requested resource=
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.throwInvalidResourceException(SchedulerUtils.java:525)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.checkResourceRequestAgainstAvailableResource(SchedulerUtils.java:415)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:349)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:304)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:312)
at
org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:268)
at
org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:254)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
at
org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:93)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:434)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:105)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:621)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:589)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048)
{noformat}
Weird part is, it passes when tested alone or tested twice (with copy&paste).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (PIG-5454) Make ParallelGC the default Garbage Collection
[ https://issues.apache.org/jira/browse/PIG-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5454: -- Attachment: pig-5454-v03.patch v02 still didn't work for Spark. It turns out spark also needed pigcontext properties to be updated. v03 uploaded. > Make ParallelGC the default Garbage Collection > -- > > Key: PIG-5454 > URL: https://issues.apache.org/jira/browse/PIG-5454 > Project: Pig > Issue Type: Bug > Components: impl >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Attachments: pig-5454-v01.patch, pig-5454-v02.patch, > pig-5454-v03.patch > > > From JDK9 and beyond, G1GC became the default GC. > I've seen our users hitting OOM after migrating to recent jdk and the issue > going away after reverting back to ParallelGC. > Maybe the GC behavior assumed by SelfSpillBag does not work with G1GC. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5454) Make ParallelGC the default Garbage Collection
[ https://issues.apache.org/jira/browse/PIG-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5454: -- Attachment: pig-5454-v02.patch Initial patch didn't work for Tez. Properties inside PigContext also needed to be updated. Uploading v02 patch. > Make ParallelGC the default Garbage Collection > -- > > Key: PIG-5454 > URL: https://issues.apache.org/jira/browse/PIG-5454 > Project: Pig > Issue Type: Bug > Components: impl >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Attachments: pig-5454-v01.patch, pig-5454-v02.patch > > > From JDK9 and beyond, G1GC became the default GC. > I've seen our users hitting OOM after migrating to recent jdk and the issue > going away after reverting back to ParallelGC. > Maybe the GC behavior assumed by SelfSpillBag does not work with G1GC. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5457) Upgrade Zookeeper to 3.7.2 (from 3.5.7)
[ https://issues.apache.org/jira/browse/PIG-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5457: -- Attachment: pig-5457-v02.patch > Upgrade Zookeeper to 3.7.2 (from 3.5.7) > --- > > Key: PIG-5457 > URL: https://issues.apache.org/jira/browse/PIG-5457 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Fix For: 0.19.0 > > Attachments: pig-5457-v01.patch, pig-5457-v02.patch > > > As mentioned in PIG-5456, zookeeper-3.5.7 dependency pulls in > log4j-1.2.17.jar that we want to avoid. Updating to 3.6.4, making it same as > the dependency from hadoop 3.3.6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5457) Upgrade Zookeeper to 3.7.2 (from 3.5.7)
[ https://issues.apache.org/jira/browse/PIG-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5457: -- Attachment: (was: pig-5457-zookeeper.patch) > Upgrade Zookeeper to 3.7.2 (from 3.5.7) > --- > > Key: PIG-5457 > URL: https://issues.apache.org/jira/browse/PIG-5457 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Fix For: 0.19.0 > > Attachments: pig-5457-v01.patch, pig-5457-v02.patch > > > As mentioned in PIG-5456, zookeeper-3.5.7 dependency pulls in > log4j-1.2.17.jar that we want to avoid. Updating to 3.6.4, making it same as > the dependency from hadoop 3.3.6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5457) Upgrade Zookeeper to 3.7.2 (from 3.5.7)
[ https://issues.apache.org/jira/browse/PIG-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5457: -- Attachment: pig-5457-zookeeper.patch Summary: Upgrade Zookeeper to 3.7.2 (from 3.5.7) (was: Upgrade Zookeeper to 3.6.4 (from 3.5.7)) Instead of 3.6, upgrading to 3.7. Tried 3.8 also but this made the tests unstable. Will re-visit in the future. Also, spark is pulling zookeeper 3.6. Skipping them. > Upgrade Zookeeper to 3.7.2 (from 3.5.7) > --- > > Key: PIG-5457 > URL: https://issues.apache.org/jira/browse/PIG-5457 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Fix For: 0.19.0 > > Attachments: pig-5457-v01.patch, pig-5457-zookeeper.patch > > > As mentioned in PIG-5456, zookeeper-3.5.7 dependency pulls in > log4j-1.2.17.jar that we want to avoid. Updating to 3.6.4, making it same as > the dependency from hadoop 3.3.6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5454) Make ParallelGC the default Garbage Collection
[ https://issues.apache.org/jira/browse/PIG-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5454: -- Attachment: pig-5454-v01.patch This was not as simple as I hoped for. I was incorrectly assuming that when multiple GCs are specified, jvm will pick the last one. Instead, jvm fails to start with bq. Conflicting collector combinations in option list; please refer to the release notes for the combinations allowed Here, attaching a patch that looks at the specified options and only adds "-XX:+UseParallelGC" when other GC is not specified. > Make ParallelGC the default Garbage Collection > -- > > Key: PIG-5454 > URL: https://issues.apache.org/jira/browse/PIG-5454 > Project: Pig > Issue Type: Bug > Components: impl >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Attachments: pig-5454-v01.patch > > > From JDK9 and beyond, G1GC became the default GC. > I've seen our users hitting OOM after migrating to recent jdk and the issue > going away after reverting back to ParallelGC. > Maybe the GC behavior assumed by SelfSpillBag does not work with G1GC. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (PIG-5454) Make ParallelGC the default Garbage Collection
[ https://issues.apache.org/jira/browse/PIG-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi reassigned PIG-5454: - Assignee: Koji Noguchi > Make ParallelGC the default Garbage Collection > -- > > Key: PIG-5454 > URL: https://issues.apache.org/jira/browse/PIG-5454 > Project: Pig > Issue Type: Bug > Components: impl >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > > From JDK9 and beyond, G1GC became the default GC. > I've seen our users hitting OOM after migrating to recent jdk and the issue > going away after reverting back to ParallelGC. > Maybe the GC behavior assumed by SelfSpillBag does not work with G1GC. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5462) Always update Owasp version to latest
[ https://issues.apache.org/jira/browse/PIG-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5462: -- Attachment: pig-5462-v02.patch Summary: Always update Owasp version to latest (was: Update Owasp version to latest (10.0.3) ) Instead of hard coding the latest version, this will always pull the latest available. Uploaded the v02 patch. bq. Like hadoop-shims-0.10.3 being reported as vulnerable. Unfortunately, this false positive remained. Reading https://nvd.nist.gov/vuln/search/results?form_type=Advanced&results_type=overview&search_type=all&cpe_vendor=cpe%3A%2F%3Aapache&cpe_product=cpe%3A%2F%3Aapache%3Ahadoop&cpe_version=cpe%3A%2F%3Aapache%3Ahadoop%3A0.10.3 it seems like it's showing the vulnerability of hadoop 0.10 version which is completely unrelated here. I'll write a separate patch for ignoring those false positives. > Always update Owasp version to latest > -- > > Key: PIG-5462 > URL: https://issues.apache.org/jira/browse/PIG-5462 > Project: Pig > Issue Type: Test >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Attachments: pig-5462-v01.patch, pig-5462-v02.patch > > > While looking at owasp report, a lot of them were completely off. > (Like hadoop-shims-0.10.3 being reported as vulnerable.) > Using latest org.owasp/dependency-check-ant > (https://mvnrepository.com/artifact/org.owasp/dependency-check-ant) > seems to help cut down the false positives. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5462) Update Owasp version to latest (10.0.3)
[ https://issues.apache.org/jira/browse/PIG-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5462: -- Attachment: pig-5462-v01.patch > Update Owasp version to latest (10.0.3) > > > Key: PIG-5462 > URL: https://issues.apache.org/jira/browse/PIG-5462 > Project: Pig > Issue Type: Test >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Attachments: pig-5462-v01.patch > > > While looking at owasp report, a lot of them were completely off. > (Like hadoop-shims-0.10.3 being reported as vulnerable.) > Using latest org.owasp/dependency-check-ant > (https://mvnrepository.com/artifact/org.owasp/dependency-check-ant) > seems to help cut down the false positives. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (PIG-5462) Update Owasp version to latest (10.0.3)
Koji Noguchi created PIG-5462: - Summary: Update Owasp version to latest (10.0.3) Key: PIG-5462 URL: https://issues.apache.org/jira/browse/PIG-5462 Project: Pig Issue Type: Test Reporter: Koji Noguchi Assignee: Koji Noguchi While looking at owasp report, a lot of them were completely off. (Like hadoop-shims-0.10.3 being reported as vulnerable.) Using latest org.owasp/dependency-check-ant (https://mvnrepository.com/artifact/org.owasp/dependency-check-ant) seems to help cut down the false positives. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5461) E2E environment variables ignored
[ https://issues.apache.org/jira/browse/PIG-5461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5461: -- Attachment: pig-5461-v01.patch > E2E environment variables ignored > - > > Key: PIG-5461 > URL: https://issues.apache.org/jira/browse/PIG-5461 > Project: Pig > Issue Type: Test >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Attachments: pig-5461-v01.patch > > > When running e2e against Hadoop3 and using hadoop2+oldpig for verification, I > was confused why environment variables like OLD_HADOOP_HOME were ignored. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (PIG-5461) E2E environment variables ignored
Koji Noguchi created PIG-5461: - Summary: E2E environment variables ignored Key: PIG-5461 URL: https://issues.apache.org/jira/browse/PIG-5461 Project: Pig Issue Type: Test Reporter: Koji Noguchi Assignee: Koji Noguchi When running e2e against Hadoop3 and using hadoop2+oldpig for verification, I was confused why environment variables like OLD_HADOOP_HOME were ignored. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5459) Jython_Checkin_3 e2e failing with NoClassDefFoundError (hadoop3)
[
https://issues.apache.org/jira/browse/PIG-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-5459:
--
Attachment: pig-5459-v01.patch
> Second option is to give it up and add the required junit jars to lib dir.
>
Attaching a patch which does this.
> Jython_Checkin_3 e2e failing with NoClassDefFoundError (hadoop3)
>
>
> Key: PIG-5459
> URL: https://issues.apache.org/jira/browse/PIG-5459
> Project: Pig
> Issue Type: Bug
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Minor
> Attachments: pig-5459-v01.patch
>
>
> {noformat}
> turing_jython.conf/Jython_Checkin_3.pig", line 4, in _module_
> from org.apache.hadoop.conf import *
> java.lang.NoClassDefFoundError: Lorg/junit/rules/ExpectedException;
> at java.lang.Class.getDeclaredFields0(Native Method)
> at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
> at java.lang.Class.privateGetPublicFields(Class.java:2614)
> at java.lang.Class.getFields(Class.java:1557)
> at org.python.core.PyJavaType.init(PyJavaType.java:419)
> at org.python.core.PyType.createType(PyType.java:1523)
> at org.python.core.PyType.addFromClass(PyType.java:1462)
> at org.python.core.PyType.fromClass(PyType.java:1551)
> at
> org.python.core.adapter.ClassicPyObjectAdapter$6.adapt(ClassicPyObjectAdapter.java:77)
> at
> org.python.core.adapter.ExtensiblePyObjectAdapter.adapt(ExtensiblePyObjectAdapter.java:44)
> at
> org.python.core.adapter.ClassicPyObjectAdapter.adapt(ClassicPyObjectAdapter.java:131)
> at org.python.core.Py.java2py(Py.java:2017)
> at org.python.core.PyJavaPackage.addClass(PyJavaPackage.java:86)
> at
> org.python.core.packagecache.PackageManager.basicDoDir(PackageManager.java:113)
> at
> org.python.core.packagecache.SysPackageManager.doDir(SysPackageManager.java:148)
> at org.python.core.PyJavaPackage.fillDir(PyJavaPackage.java:120)
> at org.python.core.imp.importAll(imp.java:1189)
> at org.python.core.imp.importAll(imp.java:1177)
> at
> org.python.pycode._pyx0.f$0(/tmp/yarn-local/usercache/.../gtrain-1722336537-turing_jython.conf/Jython_Checkin_3.pig:8)
> at
> org.python.pycode._pyx0.call_function(/tmp/yarn-local/usercache...gtrain-1722336537-tu/ring_jython.conf/Jython_Checkin_3.pig)
> at org.python.core.PyTableCode.call(PyTableCode.java:171)
> at org.python.core.PyCode.call(PyCode.java:18)
> at org.python.core.Py.runCode(Py.java:1614)
> at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:296)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:217)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:440)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:424)
> at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:310)
> at org.apache.pig.Main.runEmbeddedScript(Main.java:1096)
> at org.apache.pig.Main.run(Main.java:584)
> at org.apache.pig.Main.main(Main.java:175)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:328)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:241)
> Caused by: java.lang.ClassNotFoundException: org.junit.rules.ExpectedException
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 37 more
> java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError:
> Lorg/junit/rules/ExpectedException;
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (PIG-5460) Allow Tez to be launched from mapreduce job
[ https://issues.apache.org/jira/browse/PIG-5460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5460: -- Attachment: pig-5460-v01.patch > Allow Tez to be launched from mapreduce job > --- > > Key: PIG-5460 > URL: https://issues.apache.org/jira/browse/PIG-5460 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Priority: Minor > Attachments: pig-5460-v01.patch > > > It's like Oozie but not using Oozie launcher. > I would like to be able to submit Pig on Tez job from the mapper task. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (PIG-5460) Allow Tez to be launched from mapreduce job
[ https://issues.apache.org/jira/browse/PIG-5460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi reassigned PIG-5460: - Assignee: Koji Noguchi > Allow Tez to be launched from mapreduce job > --- > > Key: PIG-5460 > URL: https://issues.apache.org/jira/browse/PIG-5460 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Minor > Attachments: pig-5460-v01.patch > > > It's like Oozie but not using Oozie launcher. > I would like to be able to submit Pig on Tez job from the mapper task. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (PIG-5460) Allow Tez to be launched from mapreduce job
Koji Noguchi created PIG-5460: - Summary: Allow Tez to be launched from mapreduce job Key: PIG-5460 URL: https://issues.apache.org/jira/browse/PIG-5460 Project: Pig Issue Type: Improvement Reporter: Koji Noguchi It's like Oozie but not using Oozie launcher. I would like to be able to submit Pig on Tez job from the mapper task. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PIG-5459) Jython_Checkin_3 e2e failing with NoClassDefFoundError (hadoop3)
[
https://issues.apache.org/jira/browse/PIG-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17869647#comment-17869647
]
Koji Noguchi commented on PIG-5459:
---
It confused me on why regular run (e2e) is requiring junit jar.
It turns out
"from org.apache.hadoop.conf import *"
line matches classes from test jars that Hadoop3 added as part of regular lib.
For example
{noformat}
/tmp/hadoop-3.3.6/share/hadoop/common/hadoop-common-3.3.6-tests.jar ===
0 Sun Jun 18 08:22:40 UTC 2023 org/apache/hadoop/conf/
2151 Sun Jun 18 08:22:38 UTC 2023
org/apache/hadoop/conf/TestConfigurationDeprecation$1.class
522 Sun Jun 18 08:22:38 UTC 2023
org/apache/hadoop/conf/TestGetInstances$SampleClass.class
2291 Sun Jun 18 08:22:38 UTC 2023
org/apache/hadoop/conf/TestConfigurationDeprecation$2.class
333 Sun Jun 18 08:22:38 UTC 2023
org/apache/hadoop/conf/TestGetInstances$ChildInterface.class
2203 Sun Jun 18 08:22:38 UTC 2023
org/apache/hadoop/conf/TestGetInstances.class
2358 Sun Jun 18 08:22:36 UTC 2023
org/apache/hadoop/conf/TestConfigurationSubclass.class
3335 Sun Jun 18 08:22:36 UTC 2023
org/apache/hadoop/conf/TestDeprecatedKeys.class
71538 Sun Jun 18 08:22:36 UTC 2023
org/apache/hadoop/conf/TestConfiguration.class
...
/tmp/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.6-tests.jar
===
0 Sun Jun 18 08:42:34 UTC 2023 org/apache/hadoop/conf/
4469 Sun Jun 18 08:42:34 UTC 2023
org/apache/hadoop/conf/TestNoDefaultsJobConf.class
...
{noformat}
Now, these classes requires junit.
One option is to skip these test jars but that requires changes on the hadoop
side (since pig is calling hadoop commandline to start up pig.)
Second option is to give it up and add the required junit jars to lib dir.
Third option is to skip this test and let users add junit jars if they really
need to call
"from org.apache.hadoop.conf import *". but it's pretty tough to understand
what's happening when users hit this.
> Jython_Checkin_3 e2e failing with NoClassDefFoundError (hadoop3)
>
>
> Key: PIG-5459
> URL: https://issues.apache.org/jira/browse/PIG-5459
> Project: Pig
> Issue Type: Bug
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Minor
>
> {noformat}
> turing_jython.conf/Jython_Checkin_3.pig", line 4, in _module_
> from org.apache.hadoop.conf import *
> java.lang.NoClassDefFoundError: Lorg/junit/rules/ExpectedException;
> at java.lang.Class.getDeclaredFields0(Native Method)
> at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
> at java.lang.Class.privateGetPublicFields(Class.java:2614)
> at java.lang.Class.getFields(Class.java:1557)
> at org.python.core.PyJavaType.init(PyJavaType.java:419)
> at org.python.core.PyType.createType(PyType.java:1523)
> at org.python.core.PyType.addFromClass(PyType.java:1462)
> at org.python.core.PyType.fromClass(PyType.java:1551)
> at
> org.python.core.adapter.ClassicPyObjectAdapter$6.adapt(ClassicPyObjectAdapter.java:77)
> at
> org.python.core.adapter.ExtensiblePyObjectAdapter.adapt(ExtensiblePyObjectAdapter.java:44)
> at
> org.python.core.adapter.ClassicPyObjectAdapter.adapt(ClassicPyObjectAdapter.java:131)
> at org.python.core.Py.java2py(Py.java:2017)
> at org.python.core.PyJavaPackage.addClass(PyJavaPackage.java:86)
> at
> org.python.core.packagecache.PackageManager.basicDoDir(PackageManager.java:113)
> at
> org.python.core.packagecache.SysPackageManager.doDir(SysPackageManager.java:148)
> at org.python.core.PyJavaPackage.fillDir(PyJavaPackage.java:120)
> at org.python.core.imp.importAll(imp.java:1189)
> at org.python.core.imp.importAll(imp.java:1177)
> at
> org.python.pycode._pyx0.f$0(/tmp/yarn-local/usercache/.../gtrain-1722336537-turing_jython.conf/Jython_Checkin_3.pig:8)
> at
> org.python.pycode._pyx0.call_function(/tmp/yarn-local/usercache...gtrain-1722336537-tu/ring_jython.conf/Jython_Checkin_3.pig)
> at org.python.core.PyTableCode.call(PyTableCode.java:171)
> at org.python.core.PyCode.call(PyCode.java:18)
> at org.python.core.Py.runCode(Py.java:1614)
> at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:296)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:217)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:440)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:424)
> at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:310)
> at org.apache.pig.Main.runEmbeddedScript(Main.java:1096)
>
[jira] [Assigned] (PIG-5459) Jython_Checkin_3 e2e failing with NoClassDefFoundError (hadoop3)
[
https://issues.apache.org/jira/browse/PIG-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi reassigned PIG-5459:
-
Assignee: Koji Noguchi
> Jython_Checkin_3 e2e failing with NoClassDefFoundError (hadoop3)
>
>
> Key: PIG-5459
> URL: https://issues.apache.org/jira/browse/PIG-5459
> Project: Pig
> Issue Type: Bug
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Minor
>
> {noformat}
> turing_jython.conf/Jython_Checkin_3.pig", line 4, in _module_
> from org.apache.hadoop.conf import *
> java.lang.NoClassDefFoundError: Lorg/junit/rules/ExpectedException;
> at java.lang.Class.getDeclaredFields0(Native Method)
> at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
> at java.lang.Class.privateGetPublicFields(Class.java:2614)
> at java.lang.Class.getFields(Class.java:1557)
> at org.python.core.PyJavaType.init(PyJavaType.java:419)
> at org.python.core.PyType.createType(PyType.java:1523)
> at org.python.core.PyType.addFromClass(PyType.java:1462)
> at org.python.core.PyType.fromClass(PyType.java:1551)
> at
> org.python.core.adapter.ClassicPyObjectAdapter$6.adapt(ClassicPyObjectAdapter.java:77)
> at
> org.python.core.adapter.ExtensiblePyObjectAdapter.adapt(ExtensiblePyObjectAdapter.java:44)
> at
> org.python.core.adapter.ClassicPyObjectAdapter.adapt(ClassicPyObjectAdapter.java:131)
> at org.python.core.Py.java2py(Py.java:2017)
> at org.python.core.PyJavaPackage.addClass(PyJavaPackage.java:86)
> at
> org.python.core.packagecache.PackageManager.basicDoDir(PackageManager.java:113)
> at
> org.python.core.packagecache.SysPackageManager.doDir(SysPackageManager.java:148)
> at org.python.core.PyJavaPackage.fillDir(PyJavaPackage.java:120)
> at org.python.core.imp.importAll(imp.java:1189)
> at org.python.core.imp.importAll(imp.java:1177)
> at
> org.python.pycode._pyx0.f$0(/tmp/yarn-local/usercache/.../gtrain-1722336537-turing_jython.conf/Jython_Checkin_3.pig:8)
> at
> org.python.pycode._pyx0.call_function(/tmp/yarn-local/usercache...gtrain-1722336537-tu/ring_jython.conf/Jython_Checkin_3.pig)
> at org.python.core.PyTableCode.call(PyTableCode.java:171)
> at org.python.core.PyCode.call(PyCode.java:18)
> at org.python.core.Py.runCode(Py.java:1614)
> at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:296)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:217)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:440)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:424)
> at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:310)
> at org.apache.pig.Main.runEmbeddedScript(Main.java:1096)
> at org.apache.pig.Main.run(Main.java:584)
> at org.apache.pig.Main.main(Main.java:175)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:328)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:241)
> Caused by: java.lang.ClassNotFoundException: org.junit.rules.ExpectedException
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 37 more
> java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError:
> Lorg/junit/rules/ExpectedException;
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (PIG-5459) Jython_Checkin_3 e2e failing with NoClassDefFoundError (hadoop3)
Koji Noguchi created PIG-5459:
-
Summary: Jython_Checkin_3 e2e failing with NoClassDefFoundError
(hadoop3)
Key: PIG-5459
URL: https://issues.apache.org/jira/browse/PIG-5459
Project: Pig
Issue Type: Bug
Reporter: Koji Noguchi
{noformat}
turing_jython.conf/Jython_Checkin_3.pig", line 4, in _module_
from org.apache.hadoop.conf import *
java.lang.NoClassDefFoundError: Lorg/junit/rules/ExpectedException;
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
at java.lang.Class.privateGetPublicFields(Class.java:2614)
at java.lang.Class.getFields(Class.java:1557)
at org.python.core.PyJavaType.init(PyJavaType.java:419)
at org.python.core.PyType.createType(PyType.java:1523)
at org.python.core.PyType.addFromClass(PyType.java:1462)
at org.python.core.PyType.fromClass(PyType.java:1551)
at
org.python.core.adapter.ClassicPyObjectAdapter$6.adapt(ClassicPyObjectAdapter.java:77)
at
org.python.core.adapter.ExtensiblePyObjectAdapter.adapt(ExtensiblePyObjectAdapter.java:44)
at
org.python.core.adapter.ClassicPyObjectAdapter.adapt(ClassicPyObjectAdapter.java:131)
at org.python.core.Py.java2py(Py.java:2017)
at org.python.core.PyJavaPackage.addClass(PyJavaPackage.java:86)
at
org.python.core.packagecache.PackageManager.basicDoDir(PackageManager.java:113)
at
org.python.core.packagecache.SysPackageManager.doDir(SysPackageManager.java:148)
at org.python.core.PyJavaPackage.fillDir(PyJavaPackage.java:120)
at org.python.core.imp.importAll(imp.java:1189)
at org.python.core.imp.importAll(imp.java:1177)
at
org.python.pycode._pyx0.f$0(/tmp/yarn-local/usercache/.../gtrain-1722336537-turing_jython.conf/Jython_Checkin_3.pig:8)
at
org.python.pycode._pyx0.call_function(/tmp/yarn-local/usercache...gtrain-1722336537-tu/ring_jython.conf/Jython_Checkin_3.pig)
at org.python.core.PyTableCode.call(PyTableCode.java:171)
at org.python.core.PyCode.call(PyCode.java:18)
at org.python.core.Py.runCode(Py.java:1614)
at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:296)
at
org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:217)
at
org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:440)
at
org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:424)
at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:310)
at org.apache.pig.Main.runEmbeddedScript(Main.java:1096)
at org.apache.pig.Main.run(Main.java:584)
at org.apache.pig.Main.main(Main.java:175)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:328)
at org.apache.hadoop.util.RunJar.main(RunJar.java:241)
Caused by: java.lang.ClassNotFoundException: org.junit.rules.ExpectedException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 37 more
java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError:
Lorg/junit/rules/ExpectedException;
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (PIG-5459) Jython_Checkin_3 e2e failing with NoClassDefFoundError (hadoop3)
[
https://issues.apache.org/jira/browse/PIG-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-5459:
--
Priority: Minor (was: Major)
> Jython_Checkin_3 e2e failing with NoClassDefFoundError (hadoop3)
>
>
> Key: PIG-5459
> URL: https://issues.apache.org/jira/browse/PIG-5459
> Project: Pig
> Issue Type: Bug
>Reporter: Koji Noguchi
>Priority: Minor
>
> {noformat}
> turing_jython.conf/Jython_Checkin_3.pig", line 4, in _module_
> from org.apache.hadoop.conf import *
> java.lang.NoClassDefFoundError: Lorg/junit/rules/ExpectedException;
> at java.lang.Class.getDeclaredFields0(Native Method)
> at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
> at java.lang.Class.privateGetPublicFields(Class.java:2614)
> at java.lang.Class.getFields(Class.java:1557)
> at org.python.core.PyJavaType.init(PyJavaType.java:419)
> at org.python.core.PyType.createType(PyType.java:1523)
> at org.python.core.PyType.addFromClass(PyType.java:1462)
> at org.python.core.PyType.fromClass(PyType.java:1551)
> at
> org.python.core.adapter.ClassicPyObjectAdapter$6.adapt(ClassicPyObjectAdapter.java:77)
> at
> org.python.core.adapter.ExtensiblePyObjectAdapter.adapt(ExtensiblePyObjectAdapter.java:44)
> at
> org.python.core.adapter.ClassicPyObjectAdapter.adapt(ClassicPyObjectAdapter.java:131)
> at org.python.core.Py.java2py(Py.java:2017)
> at org.python.core.PyJavaPackage.addClass(PyJavaPackage.java:86)
> at
> org.python.core.packagecache.PackageManager.basicDoDir(PackageManager.java:113)
> at
> org.python.core.packagecache.SysPackageManager.doDir(SysPackageManager.java:148)
> at org.python.core.PyJavaPackage.fillDir(PyJavaPackage.java:120)
> at org.python.core.imp.importAll(imp.java:1189)
> at org.python.core.imp.importAll(imp.java:1177)
> at
> org.python.pycode._pyx0.f$0(/tmp/yarn-local/usercache/.../gtrain-1722336537-turing_jython.conf/Jython_Checkin_3.pig:8)
> at
> org.python.pycode._pyx0.call_function(/tmp/yarn-local/usercache...gtrain-1722336537-tu/ring_jython.conf/Jython_Checkin_3.pig)
> at org.python.core.PyTableCode.call(PyTableCode.java:171)
> at org.python.core.PyCode.call(PyCode.java:18)
> at org.python.core.Py.runCode(Py.java:1614)
> at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:296)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:217)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine.load(JythonScriptEngine.java:440)
> at
> org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:424)
> at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:310)
> at org.apache.pig.Main.runEmbeddedScript(Main.java:1096)
> at org.apache.pig.Main.run(Main.java:584)
> at org.apache.pig.Main.main(Main.java:175)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:328)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:241)
> Caused by: java.lang.ClassNotFoundException: org.junit.rules.ExpectedException
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 37 more
> java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError:
> Lorg/junit/rules/ExpectedException;
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (PIG-5458) Update metrics-core.version
[ https://issues.apache.org/jira/browse/PIG-5458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17869453#comment-17869453 ] Koji Noguchi commented on PIG-5458: --- Forgot to mention, after the change in PIG-5456, I noticed Pig on MR/Tez jobs were relying on metrics jar from Spark.Thus this patch. > Update metrics-core.version > > > Key: PIG-5458 > URL: https://issues.apache.org/jira/browse/PIG-5458 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Attachments: pig-5458-v01.patch > > > Hadoop3 uses metrics-core.version of 3.2.4 from io.dropwizard.metrics > and > Hadoop2 uses metrics-core.version of 3.0.1 from com.codahale.metrics. > I believe one from com.yammer.metrics (2.1.2) can be dropped. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PIG-5456) Upgrade Spark to 3.4.3
[
https://issues.apache.org/jira/browse/PIG-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17869452#comment-17869452
]
Koji Noguchi commented on PIG-5456:
---
In summary, changes to classloading is, for bin/pig and unit tests,
* MR/Tez jobs will stop using jars from spark directory.
* For Spark3, it would stop using reload4j (and orc-core after PIG-5457)
Former led to PIG-5458 where I noticed Pig on MR/Tez were relying on metrics
jar from Spark.
> Upgrade Spark to 3.4.3
> --
>
> Key: PIG-5456
> URL: https://issues.apache.org/jira/browse/PIG-5456
> Project: Pig
> Issue Type: Improvement
> Components: spark
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Major
> Fix For: 0.19.0
>
> Attachments: pig-5456-v01.patch, pig-5456-v02.patch
>
>
> Major blocker for upgrading to Spark 3.4.3 was Spark started using log4j2.
> Simple upgrade failing a lot of tests with
> {noformat}
> java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter
> overrides final method getTimeStamp.()J {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (PIG-5458) Update metrics-core.version
[ https://issues.apache.org/jira/browse/PIG-5458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5458: -- Attachment: pig-5458-v01.patch > Update metrics-core.version > > > Key: PIG-5458 > URL: https://issues.apache.org/jira/browse/PIG-5458 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Attachments: pig-5458-v01.patch > > > Hadoop3 uses metrics-core.version of 3.2.4 from io.dropwizard.metrics > and > Hadoop2 uses metrics-core.version of 3.0.1 from com.codahale.metrics. > I believe one from com.yammer.metrics (2.1.2) can be dropped. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (PIG-5458) Update metrics-core.version
[ https://issues.apache.org/jira/browse/PIG-5458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi reassigned PIG-5458: - Assignee: Koji Noguchi > Update metrics-core.version > > > Key: PIG-5458 > URL: https://issues.apache.org/jira/browse/PIG-5458 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Attachments: pig-5458-v01.patch > > > Hadoop3 uses metrics-core.version of 3.2.4 from io.dropwizard.metrics > and > Hadoop2 uses metrics-core.version of 3.0.1 from com.codahale.metrics. > I believe one from com.yammer.metrics (2.1.2) can be dropped. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (PIG-5458) Update metrics-core.version
Koji Noguchi created PIG-5458: - Summary: Update metrics-core.version Key: PIG-5458 URL: https://issues.apache.org/jira/browse/PIG-5458 Project: Pig Issue Type: Improvement Reporter: Koji Noguchi Hadoop3 uses metrics-core.version of 3.2.4 from io.dropwizard.metrics and Hadoop2 uses metrics-core.version of 3.0.1 from com.codahale.metrics. I believe one from com.yammer.metrics (2.1.2) can be dropped. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5451) Pig-on-Spark3 E2E Orc_Pushdown_5 failing
[
https://issues.apache.org/jira/browse/PIG-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-5451:
--
Attachment: pig-9-5451-v01.patch
{quote}
This was caused by conflict of orc.version.
./build/ivy/lib/Pig/orc-core-1.5.6.jar
./lib/h3/orc-core-1.5.6.jar
and
spark/jars/orc-core-1.6.14.jar
{quote}
After upgrading Spark to 3.4.3 in PIG-5456, conflict changes a bit.
When downloading spark-core 3.4.3 through ivy, no orc-core dependency.
But, when downloading spark-3.4.3-bin-without-hadoop.tgz from Apache, it
contains
orc-core-1.8.7-shaded-protobuf.jar and orc-mapreduce-1.8.7-shaded-protobuf.jar.
In order to make them consistent, adding extra pulls and adding steps to skip
orc-1.5.6 jars (just like we do with reload4j jars in PIG-5456) for Spark3.
(pig-9-5451-v01.patch)
> Pig-on-Spark3 E2E Orc_Pushdown_5 failing
> -
>
> Key: PIG-5451
> URL: https://issues.apache.org/jira/browse/PIG-5451
> Project: Pig
> Issue Type: Bug
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Minor
> Attachments: pig-9-5451-v01.patch
>
>
> Test failing with
> "java.lang.IllegalAccessError: class org.threeten.extra.chrono.HybridDate
> cannot access its superclass org.threeten.extra.chrono.AbstractDate"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (PIG-5456) Upgrade Spark to 3.4.3
[
https://issues.apache.org/jira/browse/PIG-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-5456:
--
Attachment: pig-5456-v02.patch
> log4j-1.2.17.jar was coming from stale zookeeper. Will create a new Jira to
> update the dependency.
Created PIG-5457
> As for how to skip reload4j
>
One option I considered was to move the reload4j to a different directory and
only pick it up for non-spark3 jobs. This may work if the way to start up pig
was only from bin/pig or unit/e2e tests. However, given we don't know if
users have such custom startup script(s), taking another approach. Leaving
the reload4j jar in the same location but explicitly skipping it from bin/pig
and build.xml(unit) tests. This way, only Pig-on-spark jobs are affected
leaving the rest untouched. (pig-5456-v02.patch)
> Upgrade Spark to 3.4.3
> --
>
> Key: PIG-5456
> URL: https://issues.apache.org/jira/browse/PIG-5456
> Project: Pig
> Issue Type: Improvement
> Components: spark
>Reporter: Koji Noguchi
>Assignee: Koji Noguchi
>Priority: Major
> Fix For: 0.19.0
>
> Attachments: pig-5456-v01.patch, pig-5456-v02.patch
>
>
> Major blocker for upgrading to Spark 3.4.3 was Spark started using log4j2.
> Simple upgrade failing a lot of tests with
> {noformat}
> java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter
> overrides final method getTimeStamp.()J {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (PIG-5457) Upgrade Zookeeper to 3.6.4 (from 3.5.7)
[ https://issues.apache.org/jira/browse/PIG-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5457: -- Fix Version/s: 0.19.0 > Upgrade Zookeeper to 3.6.4 (from 3.5.7) > --- > > Key: PIG-5457 > URL: https://issues.apache.org/jira/browse/PIG-5457 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Fix For: 0.19.0 > > Attachments: pig-5457-v01.patch > > > As mentioned in PIG-5456, zookeeper-3.5.7 dependency pulls in > log4j-1.2.17.jar that we want to avoid. Updating to 3.6.4, making it same as > the dependency from hadoop 3.3.6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (PIG-5457) Upgrade Zookeeper to 3.6.4 (from 3.5.7)
[ https://issues.apache.org/jira/browse/PIG-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi updated PIG-5457: -- Attachment: pig-5457-v01.patch > Upgrade Zookeeper to 3.6.4 (from 3.5.7) > --- > > Key: PIG-5457 > URL: https://issues.apache.org/jira/browse/PIG-5457 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Priority: Trivial > Attachments: pig-5457-v01.patch > > > As mentioned in PIG-5456, zookeeper-3.5.7 dependency pulls in > log4j-1.2.17.jar that we want to avoid. Updating to 3.6.4, making it same as > the dependency from hadoop 3.3.6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (PIG-5457) Upgrade Zookeeper to 3.6.4 (from 3.5.7)
[ https://issues.apache.org/jira/browse/PIG-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi reassigned PIG-5457: - Assignee: Koji Noguchi > Upgrade Zookeeper to 3.6.4 (from 3.5.7) > --- > > Key: PIG-5457 > URL: https://issues.apache.org/jira/browse/PIG-5457 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Trivial > Attachments: pig-5457-v01.patch > > > As mentioned in PIG-5456, zookeeper-3.5.7 dependency pulls in > log4j-1.2.17.jar that we want to avoid. Updating to 3.6.4, making it same as > the dependency from hadoop 3.3.6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (PIG-5457) Upgrade Zookeeper to 3.6.4 (from 3.5.7)
Koji Noguchi created PIG-5457: - Summary: Upgrade Zookeeper to 3.6.4 (from 3.5.7) Key: PIG-5457 URL: https://issues.apache.org/jira/browse/PIG-5457 Project: Pig Issue Type: Improvement Reporter: Koji Noguchi As mentioned in PIG-5456, zookeeper-3.5.7 dependency pulls in log4j-1.2.17.jar that we want to avoid. Updating to 3.6.4, making it same as the dependency from hadoop 3.3.6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (PIG-5455) Upgrade Hadoop to 3.3.6 and Tez to 0.10.3
[ https://issues.apache.org/jira/browse/PIG-5455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Noguchi resolved PIG-5455. --- Hadoop Flags: Reviewed Resolution: Fixed Thanks for the review Rohini! Committed to trunk. > Upgrade Hadoop to 3.3.6 and Tez to 0.10.3 > - > > Key: PIG-5455 > URL: https://issues.apache.org/jira/browse/PIG-5455 > Project: Pig > Issue Type: Bug >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Fix For: 0.19.0 > > Attachments: pig-5455-v01.patch > > > Latest Tez (0.10.3 and later) requires Hadoop 3.3 or later > and simple upgrade of Hadoop failing the tests with > "Implementing class java.lang.IncompatibleClassChangeError: Implementing > class" > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PIG-5428) Update hadoop2,3 and tez to recent versions
[ https://issues.apache.org/jira/browse/PIG-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863944#comment-17863944 ] Koji Noguchi commented on PIG-5428: --- > Setting tez.runtime.transfer.data-via-events.enabled to false helped but not > sure where > the problem is on. Pig? Tez? > It was due to a way how Pig uses Tez different from Hive. Hopefully handled in https://issues.apache.org/jira/browse/TEZ-4570. > Update hadoop2,3 and tez to recent versions > --- > > Key: PIG-5428 > URL: https://issues.apache.org/jira/browse/PIG-5428 > Project: Pig > Issue Type: Improvement >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Fix For: 0.18.0 > > Attachments: pig-5428-v01.patch > > > PIG-5253 hadoop3 patch is committed. > Now, updating hadoop2&3, tez and other dependent library versions. > Only testing using two different parameters. > * -Dhbaseversion=2 -Dhadoopversion=2 -Dhiveversion=1 -Dsparkversion=2 > and > * -Dhbaseversion=2 -Dhadoopversion=3 -Dhiveversion=3 -Dsparkversion=2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PIG-5455) Upgrade Hadoop to 3.3.6 and Tez to 0.10.3
[ https://issues.apache.org/jira/browse/PIG-5455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863942#comment-17863942 ] Koji Noguchi commented on PIG-5455: --- Forgot to mention, I learned that disabling of tez.runtime.transfer.data-via-events.enabled done in PIG-5428 was necessary due to a bug reported in https://issues.apache.org/jira/browse/TEZ-4570. But somehow e2e tests were still not setting this flag. Moved the disabling of tez.runtime.transfer.data-via-events.enabled from TezLauncher&TezMiniCluster to TezDagBuilder to enforce this configuration. > Upgrade Hadoop to 3.3.6 and Tez to 0.10.3 > - > > Key: PIG-5455 > URL: https://issues.apache.org/jira/browse/PIG-5455 > Project: Pig > Issue Type: Bug >Reporter: Koji Noguchi >Assignee: Koji Noguchi >Priority: Major > Fix For: 0.19.0 > > Attachments: pig-5455-v01.patch > > > Latest Tez (0.10.3 and later) requires Hadoop 3.3 or later > and simple upgrade of Hadoop failing the tests with > "Implementing class java.lang.IncompatibleClassChangeError: Implementing > class" > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
