[ 
https://issues.apache.org/jira/browse/PIG-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alagappan Maruthappan updated PIG-5442:
---------------------------------------
    External issue URL: https://github.com/apache/pig/pull/40

> Add only credentials from setStoreLocation to the Job Conf
> ----------------------------------------------------------
>
>                 Key: PIG-5442
>                 URL: https://issues.apache.org/jira/browse/PIG-5442
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Alagappan Maruthappan
>            Assignee: Alagappan Maruthappan
>            Priority: Major
>
> While testing HCatStorer with Iceberg realized Pig calls setStoreLocation on 
> all Stores with the same Job object - 
> [https://github.com/apache/pig/blob/b050a33c66fc22d648370b5c6bda04e0e51d3aa3/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java#L1081]
> Setting populated by one store is affecting the other stores. In my case the 
> "mapred.output.committer.class" is set as HiveIcebergCommitter by PigStore 
> that is used by the Iceberg table and the other stores which inserts data to 
> a non-iceberg tables also use that setting and trying to use 
> HiveIcebergCommitter.
>  
> On checking with [~rohini] , it is called to get the credentials from all 
> stores since addCredentials API was added later and not all stores have 
> implemented it and some still set configuration in setLocation method (i.e, 
> HCatStorer). 
>  
> Fixed it by passing a separate copy of Job object to each store's setLocation 
> method and adding only the credential object from the call.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to