Alagappan Maruthappan created PIG-5442:
------------------------------------------
Summary: Add only credentials from setStoreLocation to the Job Conf
Key: PIG-5442
URL: https://issues.apache.org/jira/browse/PIG-5442
Project: Pig
Issue Type: Bug
Reporter: Alagappan Maruthappan
While testing HCatStorer with Iceberg realized Pig calls setStoreLocation on
all Stores with the same Job object -
[https://github.com/apache/pig/blob/b050a33c66fc22d648370b5c6bda04e0e51d3aa3/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java#L1081]
Setting populated by one store is affecting the other stores. In my case the
"mapred.output.committer.class" is set as HiveIcebergCommitter by PigStore that
is used by the Iceberg table and the other stores which inserts data to a
non-iceberg tables also use that setting and trying to use HiveIcebergCommitter.
On checking with [~rohini] , it is called to get the credentials from all
stores since addCredentials API was added later and not all stores have
implemented it and some still set configuration in setLocation method (i.e,
HCatStorer).
Fixed it by passing a separate copy of Job object to each store's setLocation
method and adding only the credential object from the call.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
