Hi Sumit,
Since the entire cluster provisioning process must be fully automated, I had to
find "a better way", i.e. using Ambari's stack's REST endpoint, I issue (an
equivalent of)
$curl -v -i -X PUT --user admin:admin -H "X-Requested-By: ambari" --data
'{"Repositories":{"base_url":"http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.0","verify_base_url";
: false}}'
http://[AmbariServerNode]:8080/api/v1/stacks/HDP/versions/2.3/operating_systems/redhat6/repositories/HDP-2.3
(i.e. what I mentioned earlier about changing the base URL for repository used
prior to provisioning cluster from blueprint).
So, it seems it's not a bug (you are aware of this). Then stack definition
(http://[AmbariServerNode]:8080/#/main/admin/stack/services) for selected stack
(i.e. HDP 2.3) cannot be taken as "single source of truth", as there might be
other settings (as it turned out) that affect which version of given service is
actually installed.
Thank you for your input.
Regards,
Michal
________________________________
From: Sumit Mohanty <[email protected]>
Sent: Monday, January 4, 2016 8:44 PM
To: Michal Siemaszko; Alejandro Fernandez; [email protected]; Robert
Nettleton
Subject: Re: Services' versions defined in stack ignored when provisioning
cluster via blueprint
One way is to modify the repoinfo.xml file for HDP-2.3 to point to the desired
repo URL and then modifying
<latest>http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json</latest>
to point to an invalid file (e.g. invalid_hdp_urlinfo.json)
And restart Ambari.
There might be a better way.
________________________________
From: Michal Siemaszko <[email protected]>
Sent: Monday, January 04, 2016 11:12 AM
To: Alejandro Fernandez; [email protected]; Sumit Mohanty
Subject: Re: Services' versions defined in stack ignored when provisioning
cluster via blueprint
Hi Alejandro,
We use Ambari 2.1.2.
Blueprint I use to create cluster is pointed to stack "HDP", version "2.3".
Definition for that stack viewed via Ambari GUI (i.e.
http://[AmbariServerNode]:8080/#/main/admin/stack/services) has Spark v.
"1.4.1.2.3" in it.
Since we use some additional services in that cluster that are only tested for
certain versions of Spark, we need to make sure what's in stack definition is
exactly what is being installed. I did get around this problem by modifying
base URL, but I'm not sure how stack definition should be viewed then, if this
info can be changed somewhere else (in that metainfo.xml file you mentioned)
without this being reflected in Ambari GUI.
Perhaps in that blueprint we should specify stack release version as well? I.e.
instead of "2.3", use "2.3.2" (so it doesn't automatically assume latest
release version, 2.3.4 in that case)?
Thank you very much for your answer - we wanted to understand how this is
supposed to work, perhaps this is deliberate, but I've used stack definition so
far to verify services versions to be installed.
Regards,
Michal
________________________________
From: Alejandro Fernandez <[email protected]>
Sent: Monday, January 4, 2016 7:13 PM
To: [email protected]; Sumit Mohanty; Michal Siemaszko
Subject: Re: Services' versions defined in stack ignored when provisioning
cluster via blueprint
Hi Michal,
Which version of Ambari are you on?
In AMBARI-13830<https://issues.apache.org/jira/browse/AMBARI-13830> (on trunk),
Spark's metainfo.xml file contains,
<name>SPARK</name>
<version>1.5.2.2.3</version>
<extends>common-services/SPARK/1.4.1.2.3</extends>
It may very well be possible that the value is changed depending on which
version of HDP 2.3 you install.
Thanks,
Alejandro
From: Michal Siemaszko
<[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Monday, January 4, 2016 at 8:34 AM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Services' versions defined in stack ignored when provisioning cluster
via blueprint
Hi,
For a project where creation of Hadoop clusters needs to be fully automated, I
utilized Ambari's blueprints and auto-discovery features, in addition to
bootstraping hosts via REST instead of GUI (so the manual host registration is
not necessary anymore).
I run into an issue where versions of services defined in stack used are
ignored during cluster provisioning. For example, I specify "stack_name" as
"HDP" and "stack_version" as "2.3" in blueprint I use, and "Spark" version
associated with that stack is "1.4.1.2.3". However, once cluster is
provisioned, even though
http://[AmbariServerNode]:8080/#/main/admin/stack/services also shows "Spark"
version "1.4.1.2.3" installed, when submitting a sample Spark job via Spark
shell on client node, "Spark" version "1.5.2" is shown;
`/usr/hdp/2.3.4.0-3485/spark/RELEASE` file on that node also shows "Spark"
version "1.5.2.2.3.4.0".
I got around this issue by changing the base URL for repository used to
'http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.0' (instead
of "http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.0";)
prior to provisioning cluster from blueprint.
I'm wondering whether this is something expected or if it's a bug, i.e. that
service version defined in stack is ignored unless such change is made.
Shouldn't services' versions defined in stack be respected and applied
regardless of base URL? Perhaps I'm missing some other settings that should be
applied in blueprint or cluster creation template, but AFAIK it's all as per
docs/examples I read.
Thank you for your input.
Regards,
Michal Siemaszko
