[jira] [Commented] (HAWQ-1421) Improve PXF rpm package name format and dependencies
[
https://issues.apache.org/jira/browse/HAWQ-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951674#comment-15951674
]
Shivram Mani commented on HAWQ-1421:
Just to summarize the discussion within the pull request.
An apache complaint rpm will be the default result of running make rpm
If you need it to be hdp distribution compliant, make sure the HDP rpms (along
with the virtual rpms) are installed or made available in a repo.
This is to keep the dependancy generic across apache and every other hadoop
distribution.
> Improve PXF rpm package name format and dependencies
>
>
> Key: HAWQ-1421
> URL: https://issues.apache.org/jira/browse/HAWQ-1421
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: Build, PXF
>Reporter: Radar Lei
>Assignee: Shivram Mani
> Fix For: 2.2.0.0-incubating
>
>
> If we build pxf rpm package by 'make rpm', we will get below pxf packages:
> {quote}
> apache-tomcat-7.0.62-el6.noarch.rpm
> pxf-3.2.1.0-root.el6.noarch.rpm
> pxf-hbase_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hive_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-jdbc_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-json_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-service_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> {quote}
> These rpm packages have dependencies on Apache Hadoop components only, some
> other Hadoop distributes can't satisfy it. E.g. :
> {quote}
> rpm -ivh pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> error: Failed dependencies:
> pxf-service_3_2_1_0 >= 3.2.1.0 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop >= 2.7.1 is needed by pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop-mapreduce >= 2.7.1 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> {quote}
> We'd better make the rpm package name format and dependencies better.
> 1. Remove the version string like '3_2_1_0'.
> 2. Remove the user name from the build environment.
> 3. Consider do we need to include the apache-tomcat rpm package into HAWQ
> rpm release tarball.
> 4. Improve the hard code 'el6' string. (This might be optinal)
> 5. Improve the dependencies, including the dependencies between these pxf
> rpm packages.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HAWQ-1421) Improve PXF rpm package name format and dependencies
[
https://issues.apache.org/jira/browse/HAWQ-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950605#comment-15950605
]
Radar Lei commented on HAWQ-1421:
-
[~shivram] I'm fine with 'export HD=hdp'.
Please check the comments in PR, let's test and verify to make sure 'export
HD=hdp' works.
> Improve PXF rpm package name format and dependencies
>
>
> Key: HAWQ-1421
> URL: https://issues.apache.org/jira/browse/HAWQ-1421
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: Build, PXF
>Reporter: Radar Lei
>Assignee: Shivram Mani
> Fix For: 2.2.0.0-incubating
>
>
> If we build pxf rpm package by 'make rpm', we will get below pxf packages:
> {quote}
> apache-tomcat-7.0.62-el6.noarch.rpm
> pxf-3.2.1.0-root.el6.noarch.rpm
> pxf-hbase_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hive_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-jdbc_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-json_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-service_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> {quote}
> These rpm packages have dependencies on Apache Hadoop components only, some
> other Hadoop distributes can't satisfy it. E.g. :
> {quote}
> rpm -ivh pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> error: Failed dependencies:
> pxf-service_3_2_1_0 >= 3.2.1.0 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop >= 2.7.1 is needed by pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop-mapreduce >= 2.7.1 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> {quote}
> We'd better make the rpm package name format and dependencies better.
> 1. Remove the version string like '3_2_1_0'.
> 2. Remove the user name from the build environment.
> 3. Consider do we need to include the apache-tomcat rpm package into HAWQ
> rpm release tarball.
> 4. Improve the hard code 'el6' string. (This might be optinal)
> 5. Improve the dependencies, including the dependencies between these pxf
> rpm packages.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HAWQ-1421) Improve PXF rpm package name format and dependencies
[
https://issues.apache.org/jira/browse/HAWQ-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15949834#comment-15949834
]
Shivram Mani commented on HAWQ-1421:
[~rlei] if you need to build the RPM to work with hdp distribution, you will
need to set the following variable
{code}
HD=hdp
make rpm
{code}
The default behavior when you run make rpm would be to produce apache hadoop
compliant rpms.
If the user plans to run on hadoop without rpm, he can simply resort to
building from PXF source using make install as mentioned in
https://cwiki.apache.org/confluence/display/HAWQ/PXF+Build+and+Install
Are you suggesting another approach ?
> Improve PXF rpm package name format and dependencies
>
>
> Key: HAWQ-1421
> URL: https://issues.apache.org/jira/browse/HAWQ-1421
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: Build, PXF
>Reporter: Radar Lei
>Assignee: Shivram Mani
> Fix For: 2.2.0.0-incubating
>
>
> If we build pxf rpm package by 'make rpm', we will get below pxf packages:
> {quote}
> apache-tomcat-7.0.62-el6.noarch.rpm
> pxf-3.2.1.0-root.el6.noarch.rpm
> pxf-hbase_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hive_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-jdbc_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-json_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-service_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> {quote}
> These rpm packages have dependencies on Apache Hadoop components only, some
> other Hadoop distributes can't satisfy it. E.g. :
> {quote}
> rpm -ivh pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> error: Failed dependencies:
> pxf-service_3_2_1_0 >= 3.2.1.0 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop >= 2.7.1 is needed by pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop-mapreduce >= 2.7.1 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> {quote}
> We'd better make the rpm package name format and dependencies better.
> 1. Remove the version string like '3_2_1_0'.
> 2. Remove the user name from the build environment.
> 3. Consider do we need to include the apache-tomcat rpm package into HAWQ
> rpm release tarball.
> 4. Improve the hard code 'el6' string. (This might be optinal)
> 5. Improve the dependencies, including the dependencies between these pxf
> rpm packages.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HAWQ-1421) Improve PXF rpm package name format and dependencies
[
https://issues.apache.org/jira/browse/HAWQ-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948361#comment-15948361
]
Radar Lei commented on HAWQ-1421:
-
Thanks [~shivram], most changes looks great!
But for pxf dependencies, I failed to get pxf rpm installed with HDP
installed. Please verify PXF can work with these hadoop distributions.
rpm -ivh pxf-hdfs-3.2.1.0-1.el6.noarch.rpm
error: Failed dependencies:
pxf-service >= 3.2.1.0 is needed by pxf-hdfs-0:3.2.1.0-1.el6.noarch
hadoop >= 2.7.1 is needed by pxf-hdfs-0:3.2.1.0-1.el6.noarch
hadoop-mapreduce >= 2.7.1 is needed by pxf-hdfs-0:3.2.1.0-1.el6.noarch
rpm -qa |grep hdfs
hadoop_2_5_0_0_1245-hdfs-2.7.3.2.5.0.0-1245.el6.x86_64
hadoop_2_5_0_0_1245-hdfs-secondarynamenode-2.7.3.2.5.0.0-1245.el6.x86_64
ranger-hdfs-plugin-0.6.0.2.5.0.0-1245.el6.noarch
hadoop_2_5_0_0_1245-hdfs-namenode-2.7.3.2.5.0.0-1245.el6.x86_64
ranger_2_5_0_0_1245-hdfs-plugin-0.6.0.2.5.0.0-1245.el6.x86_64
hadoop_2_5_0_0_1245-hdfs-datanode-2.7.3.2.5.0.0-1245.el6.x86_64
Another thought is user might using binary hadoop without rpm installs, will
PXF works with this situation?
> Improve PXF rpm package name format and dependencies
>
>
> Key: HAWQ-1421
> URL: https://issues.apache.org/jira/browse/HAWQ-1421
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: Build, PXF
>Reporter: Radar Lei
>Assignee: Shivram Mani
> Fix For: 2.2.0.0-incubating
>
>
> If we build pxf rpm package by 'make rpm', we will get below pxf packages:
> {quote}
> apache-tomcat-7.0.62-el6.noarch.rpm
> pxf-3.2.1.0-root.el6.noarch.rpm
> pxf-hbase_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hive_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-jdbc_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-json_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-service_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> {quote}
> These rpm packages have dependencies on Apache Hadoop components only, some
> other Hadoop distributes can't satisfy it. E.g. :
> {quote}
> rpm -ivh pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> error: Failed dependencies:
> pxf-service_3_2_1_0 >= 3.2.1.0 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop >= 2.7.1 is needed by pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop-mapreduce >= 2.7.1 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> {quote}
> We'd better make the rpm package name format and dependencies better.
> 1. Remove the version string like '3_2_1_0'.
> 2. Remove the user name from the build environment.
> 3. Consider do we need to include the apache-tomcat rpm package into HAWQ
> rpm release tarball.
> 4. Improve the hard code 'el6' string. (This might be optinal)
> 5. Improve the dependencies, including the dependencies between these pxf
> rpm packages.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HAWQ-1421) Improve PXF rpm package name format and dependencies
[
https://issues.apache.org/jira/browse/HAWQ-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948210#comment-15948210
]
Shivram Mani commented on HAWQ-1421:
The following is the output on running make rpm for PXF
```
BUILD SUCCESSFUL
Total time: 48.714 secs
shivram:pxf shivram$ vim build/distributions/pxf-
pxf-3.2.1.0-shivram.el6.noarch.rpm
pxf-hbase-3.2.1.0-shivram.el6.noarch.rpm
pxf-hdfs-3.2.1.0-shivram.el6.noarch.rpm
pxf-hive-3.2.1.0-shivram.el6.noarch.rpm
pxf-jdbc-3.2.1.0-shivram.el6.noarch.rpm
pxf-json-3.2.1.0-shivram.el6.noarch.rpm
pxf-service-3.2.1.0-shivram.el6.noarch.rpm
```
When the build number is set:
```
export BUILD_NUMBER=1
BUILD SUCCESSFUL
Total time: 43.94 secs
shivram:pxf shivram$ vim build/distributions/pxf-
pxf-3.2.1.0-1.el6.noarch.rpm pxf-hbase-3.2.1.0-1.el6.noarch.rpm
pxf-hdfs-3.2.1.0-1.el6.noarch.rpm pxf-hive-3.2.1.0-1.el6.noarch.rpm
pxf-jdbc-3.2.1.0-1.el6.noarch.rpm pxf-json-3.2.1.0-1.el6.noarch.rpm
pxf-service-3.2.1.0-1.el6.noarch.rpm
```
> Improve PXF rpm package name format and dependencies
>
>
> Key: HAWQ-1421
> URL: https://issues.apache.org/jira/browse/HAWQ-1421
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: Build, PXF
>Reporter: Radar Lei
>Assignee: Shivram Mani
> Fix For: 2.2.0.0-incubating
>
>
> If we build pxf rpm package by 'make rpm', we will get below pxf packages:
> {quote}
> apache-tomcat-7.0.62-el6.noarch.rpm
> pxf-3.2.1.0-root.el6.noarch.rpm
> pxf-hbase_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hive_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-jdbc_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-json_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-service_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> {quote}
> These rpm packages have dependencies on Apache Hadoop components only, some
> other Hadoop distributes can't satisfy it. E.g. :
> {quote}
> rpm -ivh pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> error: Failed dependencies:
> pxf-service_3_2_1_0 >= 3.2.1.0 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop >= 2.7.1 is needed by pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop-mapreduce >= 2.7.1 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> {quote}
> We'd better make the rpm package name format and dependencies better.
> 1. Remove the version string like '3_2_1_0'.
> 2. Remove the user name from the build environment.
> 3. Consider do we need to include the apache-tomcat rpm package into HAWQ
> rpm release tarball.
> 4. Improve the hard code 'el6' string. (This might be optinal)
> 5. Improve the dependencies, including the dependencies between these pxf
> rpm packages.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HAWQ-1421) Improve PXF rpm package name format and dependencies
[
https://issues.apache.org/jira/browse/HAWQ-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948176#comment-15948176
]
Shivram Mani commented on HAWQ-1421:
[~rlei] this is good feedback.
We can remove the version information from the rpm name.
With regards to the username, we will resort to using systems user.name only if
the BUILD_NUMBER variable isn't set. Ideally the BUILD_NUMBER needs to be set.
As a default i think it is fine if we stick to the username, otherwise we will
have to default to a hardcoded value of 1 which isn't good.
We do need to include the apache tomcat rpm as this is not publicly available.
tomcat is only distributed via source
I don't think we need to change any pxf rpm dependancies. Having dependancy on
hadoop rpms works with all well known hadoop distributions
> Improve PXF rpm package name format and dependencies
>
>
> Key: HAWQ-1421
> URL: https://issues.apache.org/jira/browse/HAWQ-1421
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: Build, PXF
>Reporter: Radar Lei
>Assignee: Shivram Mani
> Fix For: 2.2.0.0-incubating
>
>
> If we build pxf rpm package by 'make rpm', we will get below pxf packages:
> {quote}
> apache-tomcat-7.0.62-el6.noarch.rpm
> pxf-3.2.1.0-root.el6.noarch.rpm
> pxf-hbase_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-hive_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-jdbc_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-json_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> pxf-service_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> {quote}
> These rpm packages have dependencies on Apache Hadoop components only, some
> other Hadoop distributes can't satisfy it. E.g. :
> {quote}
> rpm -ivh pxf-hdfs_3_2_1_0-3.2.1.0-root.el6.noarch.rpm
> error: Failed dependencies:
> pxf-service_3_2_1_0 >= 3.2.1.0 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop >= 2.7.1 is needed by pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> hadoop-mapreduce >= 2.7.1 is needed by
> pxf-hdfs_3_2_1_0-0:3.2.1.0-root.el6.noarch
> {quote}
> We'd better make the rpm package name format and dependencies better.
> 1. Remove the version string like '3_2_1_0'.
> 2. Remove the user name from the build environment.
> 3. Consider do we need to include the apache-tomcat rpm package into HAWQ
> rpm release tarball.
> 4. Improve the hard code 'el6' string. (This might be optinal)
> 5. Improve the dependencies, including the dependencies between these pxf
> rpm packages.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
