Ambari 2.7 cannot create ranger repositories for components such as hdfs.
The error is:
2018-07-27 21:58:24,712 [http-bio-6080-exec-9] INFO
org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:345)
- Request failed. loginId=amb_ranger_admin, logMessage=User is not allowed
to access the API
javax.ws.rs.WebApplicationException
at
org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:337)
at
org.apache.ranger.security.context.RangerPreAuthSecurityHandler.isAPISpnegoAccessible(RangerPreAuthSecuri
tyHandler.java:106)
My settings are:
"ranger-env": {
"properties_attributes": {},
"properties": {
"admin_username": "admin",
"admin_password": "%SERVICE_PASSWORD%",
"bind_anonymous": "false",
"create_db_dbuser": "false",
"is_solrCloud_enabled": "true",
"keyadmin_user_password": "%SERVICE_PASSWORD%",
"rangertagsync_user_password": "%SERVICE_PASSWORD%",
"rangerusersync_user_password": "%SERVICE_PASSWORD%",
"ranger_admin_username": "amb_ranger_admin",
"ranger_admin_password": "%SERVICE_PASSWORD%",
"ranger-hbase-plugin-enabled": "Yes",
"ranger-hdfs-plugin-enabled": "Yes",
"ranger-hive-plugin-enabled": "Yes",
"ranger-kafka-plugin-enabled": "Yes",
"ranger-knox-plugin-enabled": "Yes",
"ranger-storm-plugin-enabled": "Yes",
"ranger-yarn-plugin-enabled": "Yes",
"ranger_group": "ranger",
"ranger_user": "ranger",
"ranger_privelege_user_jdbc_url": "jdbc:oracle:thin:@
//%ORACEL_DB_HOST%",
"xasecure.audit.destination.hdfs": "true",
"xasecure.audit.destination.hdfs.dir":
"hdfs://%ENV%-cluster/ranger/audit",
"xasecure.audit.destination.solr": "true",
"xasecure.audit.destination.solr.zookeepers":
"%ENV%-namenode.%SUBNET%.%VCN%.oraclevcn.com:2181/infra-solr"
}
I see amb_ranger_admin is user role instead of admin role in Ranger admin
UI. However, the same settings worked in HDP2.6. What changed in HDP3.0 so
that amb_ranger_admin cannot create repositories? Appreciate any clue.
On Thu, Jul 26, 2018 at 12:30 PM, Lian Jiang <[email protected]> wrote:
> Thanks. I don't have hive installed in HDP2.6 which spark 2.2.0. I don't
> know spark2.3.1 introduces hard dependency on hive and I did not find such
> info in spark2.3.1 document.
>
> I will try install hive on my HDP3.0 cluster.
>
> On Thu, Jul 26, 2018 at 12:21 PM, Vitaly Brodetskyi <
> [email protected]> wrote:
>
>> Hi Lian Jiang
>>
>>
>> According to stack trace from SPARK2 service, looks like you don't
>> have HIVE service installed on your cluster. Because if HIVE service
>> installed, "hive-env" config should be available for sure. As i know HIVE
>> is required service for SPARK2.
>>
>>
>> Regards
>>
>> Vitalyi
>> ------------------------------
>> *Від:* Lian Jiang <[email protected]>
>> *Надіслано:* 26 липня 2018 р. 22:08
>> *Кому:* [email protected]
>> *Тема:* Re: install HDP3.0 using ambari
>>
>> During migration, spark2 and livy2-server fail to start due to:
>>
>> 2018-07-26 18:18:09,024 - The 'livy2-server' component did not advertise a
>> version. This may indicate a problem with the component packaging. However,
>> the stack-select tool was able to report a single version installed
>> (3.0.0.0-1634). This is the version that will be reported.
>> Traceback (most recent call last):
>> File
>> "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/livy2_server.py",
>> line 148, in <module>
>> LivyServer().execute()
>> File
>> "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py",
>> line 353, in execute
>> method(env)
>> File
>> "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/livy2_server.py",
>> line 43, in install
>> import params
>> File
>> "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/params.py",
>> line 220, in <module>
>> if hive_metastore_db_type == "mssql":
>> File
>> "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py",
>> line 73, in __getattr__
>> raise Fail("Configuration parameter '" + self.name + "' was not found in
>> configurations dictionary!")
>> resource_management.core.exceptions.Fail: Configuration parameter 'hive-env'
>> was not found in configurations dictionary!
>>
>> I observed that:
>> /var/lib/ambari-agent/cache/stacks/HDP/2.6/services/YARN/configuration/yarn-site.xml
>> does have:
>>
>> <property>
>> <name>yarn.nodemanager.kill-escape.user</name>
>> <value>hive</value>
>> <depends-on>
>> <property>
>> <type>hive-env</type>
>> <name>hive_user</name>
>> </property>
>> </depends-on>
>> <on-ambari-upgrade add="false"/>
>> </property>
>>
>> /var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/configuration/yarn-site.xml
>> doesn't.
>>
>> Below files are the same:
>>
>> /var/lib/ambari-agent/cache/stacks/HDP/2.6/services/SPARK2/configuration/livy2-env.xml
>>
>> vs
>> /var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/configuration/livy2-env.xml
>>
>>
>>
>> /var/lib/ambari-agent/cache/stacks/HDP/2.6/services/SPARK2/configuration/livy2-conf.xml
>>
>> vs
>> /var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/configuration/livy2-conf.xml
>>
>>
>> I don't see anything wrong in
>> /var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/configuration/spark2-defaults.xml
>> /var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/configuration/spark2-env.xml
>>
>> either.
>>
>>
>> Any idea will be highly appreciated! Thanks.
>>
>>
>> On Tue, Jul 24, 2018 at 3:56 PM, Lian Jiang <[email protected]>
>> wrote:
>>
>>> Thanks. I will try 1 given that I cannot find enough documents/examples
>>> online for the blueprint schema changes online.
>>>
>>> On Tue, Jul 24, 2018 at 3:49 PM, Benoit Perroud <[email protected]>
>>> wrote:
>>>
>>>> HDP 3 don’t have any more spark (1.x), only spark2.
>>>>
>>>> In general, old blueprints are not fully compatible and have to be
>>>> tweaked a bit.
>>>>
>>>> I see two options from where you are:
>>>>
>>>> 1) Upgrade your current blueprint, i.e. use it with HDP 2.6+, run the
>>>> upgrade wizard from Ambari 2.7 to HDP 3, and export a new version of the
>>>> blueprint.
>>>> 2) Manually update the blueprint and remove the spark-defaults section
>>>> it has. This is still not giving you the guarantee the blueprint will work,
>>>> you might need to do more customisation.
>>>>
>>>> Benoit
>>>>
>>>>
>>>>
>>>>
>>>> On 25 Jul 2018, at 00:05, Lian Jiang <[email protected]> wrote:
>>>>
>>>> Thanks Benoit for the advice.
>>>>
>>>> I switched to ambari 2.7. However, when I create the cluster, it failed
>>>> due to "config types are not defined in the stack: [spark-defaults]".
>>>>
>>>> Below links point to a spec < ambari2.7.
>>>> https://cwiki.apache.org/confluence/display/AMBARI/Blueprint
>>>> s#Blueprints-BlueprintStructure
>>>> https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.0.0/adm
>>>> inistering-ambari/content/amb_using_ambari_blueprints.html
>>>>
>>>> https://github.com/apache/ambari/tree/release-2.7.0/ambari-s
>>>> erver/src/main/resources/stacks/HDP does not have HDP3.0. This makes
>>>> it hard to troubleshoot.
>>>>
>>>> Do you know where I can find the source code of HDP3.0 ambari stack so
>>>> that I can check what configs are supported in new ambari?
>>>>
>>>> Thanks.
>>>>
>>>>
>>>>
>>>> On Mon, Jul 23, 2018 at 2:35 PM, Benoit Perroud <[email protected]>
>>>> wrote:
>>>>
>>>>> Are you using Ambari 2.7?
>>>>>
>>>>> Make sure you upgrade Ambari to 2.7 first, since this version is
>>>>> required for HDP 3
>>>>>
>>>>> Benoit
>>>>>
>>>>>
>>>>> On 23 Jul 2018, at 23:32, Lian Jiang <[email protected]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am using ambari blueprint to install HDP 3.0 and cannot register the
>>>>> vdf file.
>>>>>
>>>>> The vdf file is (the url works):
>>>>>
>>>>> {
>>>>> "VersionDefinition": {
>>>>> "version_url": "http://public-repo-1.hortonwo
>>>>> rks.com/HDP/centos7/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml"
>>>>> }
>>>>> }
>>>>>
>>>>> The error is "An internal system exception occurred: Stack data, Stack
>>>>> HDP 3.0 is not found in Ambari metainfo"
>>>>>
>>>>> Any idea? Thanks.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
