[ https://issues.apache.org/jira/browse/IMPALA-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841577#comment-17841577 ]
XiangYang commented on IMPALA-10848: ------------------------------------ Sorry I gave up it, you can continue on it. > Provide compile-only option to skip downloading test dependencies > ----------------------------------------------------------------- > > Key: IMPALA-10848 > URL: https://issues.apache.org/jira/browse/IMPALA-10848 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure > Reporter: Quanlong Huang > Assignee: XiangYang > Priority: Major > Attachments: pywebhdfs_failure.png > > > Compiling Impala is not easy for a beginner. A portion of failures are in > downloading/installing dependencies. > For instance, old versions of Impala may fail to compile since cdh components > of old GBNs on S3 are removed. However, the artifacts of cdh component are > only used in testing (minicluster & holding testdata). We can still compile > without them. > Take pip dependencies as another example, here is a failure I got from a > community user. It failed by installing pywebhdfs: > !pywebhdfs_failure.png! > However, simple git-grep shows that pywebhdfs is only used in tests: > {code:bash} > $ git grep pywebhdfs > bin/bootstrap_system.sh:# >>> from pywebhdfs.webhdfs import PyWebHdfsClient > infra/python/deps/requirements.txt:pywebhdfs == 0.3.2 > tests/common/impala_test_suite.py: # HDFS: uses a mixture of pywebhdfs > (which is faster than the HDFS CLI) and the > tests/util/hdfs_util.py:from pywebhdfs.webhdfs import PyWebHdfsClient, > errors, _raise_pywebhdfs_exception > tests/util/hdfs_util.py: > _raise_pywebhdfs_exception(response.status_code, response.text) > tests/util/hdfs_util.py: > _raise_pywebhdfs_exception(response.status_code, response.text) > tests/util/hdfs_util.py: > _raise_pywebhdfs_exception(response.status_code, response.text) > tests/util/hdfs_util.py: > _raise_pywebhdfs_exception(response.status_code, response.text) {code} > If the user just wants to compile Impala and deploys it in their existing > Hadoop cluster, dealing with these failures is a waste of their time. > *Target for this JIRA* > * Provide compile-only option to bin/bootstrap_system.sh. It should skip > downloading/installing unused dependencies like postgresql. > * Provide compile-only option to buildall.sh. It should skip downloading > unused cdh/cdp components in compilation. > * Update our > [wiki|https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala] > about this. > Note that we already have some env vars to control the download behaviors, > e.g. SKIP_PYTHON_DOWNLOAD, SKIP_TOOLCHAIN_BOOTSTRAP. We just need to make the > compile-only scenario works with minimal requirements and document it. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org