[
https://issues.apache.org/jira/browse/PIG-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alagappan Maruthappan updated PIG-5442:
---------------------------------------
External issue URL: https://github.com/apache/pig/pull/40
> Add only credentials from setStoreLocation to the Job Conf
> ----------------------------------------------------------
>
> Key: PIG-5442
> URL: https://issues.apache.org/jira/browse/PIG-5442
> Project: Pig
> Issue Type: Bug
> Reporter: Alagappan Maruthappan
> Assignee: Alagappan Maruthappan
> Priority: Major
>
> While testing HCatStorer with Iceberg realized Pig calls setStoreLocation on
> all Stores with the same Job object -
> [https://github.com/apache/pig/blob/b050a33c66fc22d648370b5c6bda04e0e51d3aa3/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java#L1081]
> Setting populated by one store is affecting the other stores. In my case the
> "mapred.output.committer.class" is set as HiveIcebergCommitter by PigStore
> that is used by the Iceberg table and the other stores which inserts data to
> a non-iceberg tables also use that setting and trying to use
> HiveIcebergCommitter.
>
> On checking with [~rohini] , it is called to get the credentials from all
> stores since addCredentials API was added later and not all stores have
> implemented it and some still set configuration in setLocation method (i.e,
> HCatStorer).
>
> Fixed it by passing a separate copy of Job object to each store's setLocation
> method and adding only the credential object from the call.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
