[
https://issues.apache.org/jira/browse/HADOOP-15708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated HADOOP-15708:
------------------------------------
Description:
Hadoop Common contains a widely used Configuration class.
This class can handle deprecations of properties, e.g. if property 'A' gets
deprecated with an alternative property key 'B', users can access property
values with keys 'A' and 'B'.
Unfortunately, this does not work in one case.
When a config file is specified (for instance, XML) and a property is read with
the config.get() method, the config is loaded from the file at this time.
If the deprecation mapping is not yet specified by the time any config value is
retrieved and the XML config refers to a deprecated key, then the deprecation
mapping specified, the config value cannot be retrieved neither with the
deprecated nor with the new key.
The attached patch contains a testcase that reproduces this wrong behavior.
Here are the steps outlined what the testcase does:
1. Creates an XML config file with a deprecated property
2. Adds the config to the Configuration object
3. Retrieves the config with its deprecated key (it does not really matter
which property the user gets, could be any)
4. Specifies the deprecation rules including the one defined in the config
5. Prints and asserts the property retrieved from the config with both the
deprecated and the new property keys.
For reference, here is the log of one execution that actually shows what the
issue is:
{noformat}
Loaded items: 1
Looked up property value with name hadoop.zk.address: null
Looked up property value with name yarn.resourcemanager.zk-address:
dummyZkAddress
Contents of config file: [<?xml version="1.0"?>, <configuration>,
<property><name>yarn.resourcemanager.zk-address</name><value>dummyZkAddress</value></property>,
</configuration>]
Looked up property value with name hadoop.zk.address: null
2018-08-31 10:10:06,484 INFO Configuration.deprecation
(Configuration.java:logDeprecation(1397)) - yarn.resourcemanager.zk-address is
deprecated. Instead, use hadoop.zk.address
Looked up property value with name hadoop.zk.address: null
Looked up property value with name hadoop.zk.address: null
java.lang.AssertionError:
Expected :dummyZkAddress
Actual :null
{noformat}
*As it's visible from the output and the code, the issue is really that if the
config is retrieved either with the deprecated or the new value, Configuration
both wants to serve the value with the new key.
If the mapping is not specified before any retrieval happened, the value is
only stored under the deprecated key but not the new key.*
was:
Hadoop Common contains a widely used Configuration class.
This class can handle deprecations of properties, e.g. if property 'A' gets
deprecated with an alternative property key 'B', users can access property
values with keys 'A' and 'B'.
Unfortunately, this does not work in one case.
When a config file is specified (for instance, XML) and a property is read with
the config.get() method, the config is loaded from the file at this time.
If the deprecation mapping is not yet specified by the time any config value is
retrieved and the XML config refers to a deprecated key, then the deprecation
mapping specified, the config value cannot be retrieved neither with the
deprecated nor with the new key.
The attached patch contains a testcase that reproduces this wrong behavior.
Here are the steps outlined what the testcase does:
1. Creates an XML config file with a deprecated property
2. Adds the config to the Configuration object
3. Retrieves the config with its deprecated key (it does not really matter
which property the user gets, could be any)
4. Specifies the deprecation rules including the one defined in the config
5. Prints and asserts the property retrieved from the config with both the
deprecated and the new property keys.
For reference, here is the log of one execution that actually shows what the
issue is:
{noformat}
Loaded items: 1
Looked up property value with name hadoop.zk.address: null
Looked up property value with name yarn.resourcemanager.zk-address:
dummyZkAddress
Contents of config file: [<?xml version="1.0"?>, <configuration>,
<property><name>yarn.resourcemanager.zk-address</name><value>dummyZkAddress</value></property>,
</configuration>]
Looked up property value with name hadoop.zk.address: null
2018-08-31 10:10:06,484 INFO Configuration.deprecation
(Configuration.java:logDeprecation(1397)) - yarn.resourcemanager.zk-address is
deprecated. Instead, use hadoop.zk.address
Looked up property value with name hadoop.zk.address: null
Looked up property value with name hadoop.zk.address: null
java.lang.AssertionError:
Expected :dummyZkAddress
Actual :null
{noformat}
*As it's visible from the output and the code, the issue is really that if the
config is retrieved either with the deprecated or the new value, Configuration
both wants to serve the value with the new key.
If the mapping is not specified before any retrieval happened, the value is
only stored under the deprecated key but not the new key. *
> Reading values from Configuration before adding deprecations make it
> impossible to read value with deprecated key
> -----------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-15708
> URL: https://issues.apache.org/jira/browse/HADOOP-15708
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Szilard Nemeth
> Assignee: Szilard Nemeth
> Priority: Major
> Attachments: HADOOP-15708-testcase.patch
>
>
> Hadoop Common contains a widely used Configuration class.
> This class can handle deprecations of properties, e.g. if property 'A' gets
> deprecated with an alternative property key 'B', users can access property
> values with keys 'A' and 'B'.
> Unfortunately, this does not work in one case.
> When a config file is specified (for instance, XML) and a property is read
> with the config.get() method, the config is loaded from the file at this
> time.
> If the deprecation mapping is not yet specified by the time any config value
> is retrieved and the XML config refers to a deprecated key, then the
> deprecation mapping specified, the config value cannot be retrieved neither
> with the deprecated nor with the new key.
> The attached patch contains a testcase that reproduces this wrong behavior.
> Here are the steps outlined what the testcase does:
> 1. Creates an XML config file with a deprecated property
> 2. Adds the config to the Configuration object
> 3. Retrieves the config with its deprecated key (it does not really matter
> which property the user gets, could be any)
> 4. Specifies the deprecation rules including the one defined in the config
> 5. Prints and asserts the property retrieved from the config with both the
> deprecated and the new property keys.
> For reference, here is the log of one execution that actually shows what the
> issue is:
> {noformat}
> Loaded items: 1
> Looked up property value with name hadoop.zk.address: null
> Looked up property value with name yarn.resourcemanager.zk-address:
> dummyZkAddress
> Contents of config file: [<?xml version="1.0"?>, <configuration>,
> <property><name>yarn.resourcemanager.zk-address</name><value>dummyZkAddress</value></property>,
> </configuration>]
> Looked up property value with name hadoop.zk.address: null
> 2018-08-31 10:10:06,484 INFO Configuration.deprecation
> (Configuration.java:logDeprecation(1397)) - yarn.resourcemanager.zk-address
> is deprecated. Instead, use hadoop.zk.address
> Looked up property value with name hadoop.zk.address: null
> Looked up property value with name hadoop.zk.address: null
> java.lang.AssertionError:
> Expected :dummyZkAddress
> Actual :null
> {noformat}
> *As it's visible from the output and the code, the issue is really that if
> the config is retrieved either with the deprecated or the new value,
> Configuration both wants to serve the value with the new key.
> If the mapping is not specified before any retrieval happened, the value is
> only stored under the deprecated key but not the new key.*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
