[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[ https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030282#comment-17030282 ] ASF subversion and git services commented on IMPALA-7006: - Commit 19a4d8fe794c9b17e69d6c65473f9a68084916bb in impala's branch refs/heads/master from Thomas Tauber-Marshall [ https://gitbox.apache.org/repos/asf?p=impala.git;h=19a4d8f ] IMPALA-9335 (part 2): Fix rebased KRPC to compile This patch applies various fixes to Impala and to the copied Kudu source code in be/src/kudu/* to allow everything to compile. Some highlights of the changes made: - Various Kudu files were removed from compilation due to issues like relying on libraries that Impala does not provide. The linking of some executable is also changed for similar reasons. - The Kudu Cache implementation changed to support unique_ptr, allowing us to remove various uses of MakeScopeExitTrigger. - Some flags that have a DEFINE in both Kudu and Impala are modified to change one of the DEFINEs to a DECLARE. This patch was in part based on the patches that were applied the last time we rebased the Kudu code in IMPALA-7006, and I ensured that all changes from those commits that are still relevant were included here. I also went through all commits that have been applied to the be/src/kudu directory since the last rebase and ensured that all relevant changes from those are included here. Testing: - Passed an exhaustive DEBUG build and a core ASAN build. Change-Id: I1eb4caf927c729109426fb50a28b5e15d6ac46cb Reviewed-on: http://gerrit.cloudera.org:8080/15144 Tested-by: Impala Public Jenkins Reviewed-by: Joe McDonnell > Rebase KRPC onto Kudu upstream repository > - > > Key: IMPALA-7006 > URL: https://issues.apache.org/jira/browse/IMPALA-7006 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Lars Volker >Assignee: Lars Volker >Priority: Major > Labels: krpc > Fix For: Impala 2.13.0, Impala 3.1.0 > > > We should consider rebasing our KRPC code on top of the latest Kudu upstream > version. This will keep the two projects more in sync and will allow us to > make use of recent improvements, e.g. around thread stack collection, without > having to pick individual changes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[ https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810074#comment-16810074 ] ASF subversion and git services commented on IMPALA-7006: - Commit 0c2d3c74d662d32eaf5c56cdeca067285ab1d300 in impala's branch refs/heads/2.x from Lars Volker [ https://gitbox.apache.org/repos/asf?p=impala.git;h=0c2d3c7 ] IMPALA-7006: Remove KRPC folders Change-Id: Ic677484c27ed18b105da0a6b0901df4eb9f248e6 Reviewed-on: http://gerrit.cloudera.org:8080/10756 Reviewed-by: Lars Volker Tested-by: Lars Volker > Rebase KRPC onto Kudu upstream repository > - > > Key: IMPALA-7006 > URL: https://issues.apache.org/jira/browse/IMPALA-7006 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Lars Volker >Assignee: Lars Volker >Priority: Major > Labels: krpc > Fix For: Impala 3.1.0 > > > We should consider rebasing our KRPC code on top of the latest Kudu upstream > version. This will keep the two projects more in sync and will allow us to > make use of recent improvements, e.g. around thread stack collection, without > having to pick individual changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[ https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810078#comment-16810078 ] ASF subversion and git services commented on IMPALA-7006: - Commit 23a3ef7452ade42a426502e0fd3719f3836d6730 in impala's branch refs/heads/2.x from Lars Volker [ https://gitbox.apache.org/repos/asf?p=impala.git;h=23a3ef7 ] IMPALA-7006: [KSECURITY] Update security library integration This commit is part of a set of changes for IMPALA-7006. It started based on an original change (f51c4435), which integrated Kudu's security folder into our build. This change removes several compile time checks that are now either done in Kudu's own cmake files or that can be removed due to Impala deprecating support for older OS versions in the 3.x line. The removed checks are: HAVE_KRB5_GET_INIT_CREDS_OPT_SET_OUT_CCACHE: We now check for this in Kudu's code. HAVE_KRB5_IS_CONFIG_PRINCIPAL, HAVE_KRB5_GET_INIT_CREDS_OPT_SET_FAST_CCACHE_NAME: These checks are not needed anymore. All OS versions supported by Impala now have sufficiently recent versions of Kerberos. Change-Id: Ifab51d887f5e771ad62eeddc14b9c47f42c3130d Reviewed-on: http://gerrit.cloudera.org:8080/10759 Reviewed-by: Lars Volker Tested-by: Lars Volker > Rebase KRPC onto Kudu upstream repository > - > > Key: IMPALA-7006 > URL: https://issues.apache.org/jira/browse/IMPALA-7006 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Lars Volker >Assignee: Lars Volker >Priority: Major > Labels: krpc > Fix For: Impala 3.1.0 > > > We should consider rebasing our KRPC code on top of the latest Kudu upstream > version. This will keep the two projects more in sync and will allow us to > make use of recent improvements, e.g. around thread stack collection, without > having to pick individual changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[
https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810085#comment-16810085
]
ASF subversion and git services commented on IMPALA-7006:
-
Commit 315bc66bbac8715302d455d2d746981cebf74aec in impala's branch
refs/heads/2.x from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=315bc66 ]
KUDU-2305: Limit sidecars to INT_MAX and fortify socket code
NOTE: This commit is part of a set of changes for IMPALA-7006. It
contains pieces of a previous commit that need to be cherry picked
again after rebasing the code in be/src/kudu/{util,security,rpc}.
The original commit message is below:
Inspection of the code revealed some other local variables
that could overflow with large messages. This patch takes
two approaches to eliminate the issues.
First, it limits the total size of the messages by limiting
the total size of the sidecars to INT_MAX. The total size
of the protobuf and header components of the message
should be considerably smaller, so limiting the sidecars
to INT_MAX eliminates messages that are larger than UINT_MAX.
This also means that the sidecar offsets, which are unsigned
32-bit integers, are also safe. Given that
FLAGS_rpc_max_message_size is limited to INT_MAX at startup,
the receiver would reject any message this large anyway.
This also helps with the networking codepath, as any given
sidecar will have a size less than INT_MAX, so every Slice
that interacts with Writev() is shorter than INT_MAX.
Second, even with sidecars limited to INT_MAX, the headers
and protobuf parts of the messages mean that certain messages
could still exceed INT_MAX. This patch changes some of the sockets
codepath to tolerate iovec's that reference more than INT_MAX
bytes total. Specifically, it changes Writev()'s nwritten bytes
to an int64_t for both TlsSocket and Socket. TlsSocket works
because it is sending each Slice individually. The first change
limited any given Slice to INT_MAX, so each individual Write()
should not be impacted. For Socket, Writev() uses sendmsg(). It
should do partial network sends to handle this case. Any Write()
call specifies its size with a 32-bit integer, and that will
not be impacted by this patch.
Testing:
- Modified TestRpcSidecarLimits() to verify that sidecars are
limited to INT_MAX bytes.
- Added a test mode to TestRpcSidecarLimits() where it
overrides rpc_max_message_size and sends the maximal
message. This verifies that the client send codepath
can handle the maximal message.
Reviewed-on: http://gerrit.cloudera.org:8080/9601
Reviewed-by: Todd Lipcon
Tested-by: Todd Lipcon
Changes from Kudu version:
- Updated declaration of FLAGS_rpc_max_message_size
in rpc-mgr.cc and added a warning not to set it
larger than INT_MAX.
Change-Id: Id23e518995f2bf2f6bf6b49d5f413f3eaa4e79d1
Reviewed-on: http://gerrit.cloudera.org:8080/9748
Reviewed-by: Michael Ho
Tested-by: Impala Public Jenkins
Reviewed-on: http://gerrit.cloudera.org:8080/10765
Reviewed-by: Lars Volker
Tested-by: Lars Volker
> Rebase KRPC onto Kudu upstream repository
> -
>
> Key: IMPALA-7006
> URL: https://issues.apache.org/jira/browse/IMPALA-7006
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Lars Volker
>Priority: Major
> Labels: krpc
> Fix For: Impala 3.1.0
>
>
> We should consider rebasing our KRPC code on top of the latest Kudu upstream
> version. This will keep the two projects more in sync and will allow us to
> make use of recent improvements, e.g. around thread stack collection, without
> having to pick individual changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[ https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810086#comment-16810086 ] ASF subversion and git services commented on IMPALA-7006: - Commit b65dbf8e40d7c8f77db05846d84497824d6bbd26 in impala's branch refs/heads/2.x from Lars Volker [ https://gitbox.apache.org/repos/asf?p=impala.git;h=b65dbf8 ] IMPALA-7006: Pick parts of recent Kudu gutil changes - Include some ASAN macros from gutil (Kudu commit c8724c61) - Pick parts of KUDU-2427 (Kudu commit b7cf3b2e) - Rename constants (Kudu commit e719b5ef) These changes will be subsumed by a proper rebase of GUTIL. Change-Id: Id2dc8c70425e3ac030427ebeb1ec18a44d14d5cb Reviewed-on: http://gerrit.cloudera.org:8080/10769 Tested-by: Impala Public Jenkins Reviewed-by: Lars Volker > Rebase KRPC onto Kudu upstream repository > - > > Key: IMPALA-7006 > URL: https://issues.apache.org/jira/browse/IMPALA-7006 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Lars Volker >Assignee: Lars Volker >Priority: Major > Labels: krpc > Fix For: Impala 3.1.0 > > > We should consider rebasing our KRPC code on top of the latest Kudu upstream > version. This will keep the two projects more in sync and will allow us to > make use of recent improvements, e.g. around thread stack collection, without > having to pick individual changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[
https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810075#comment-16810075
]
ASF subversion and git services commented on IMPALA-7006:
-
Commit dfb9e16960f858e1dccd209e7b1f7e4be60bc6d4 in impala's branch
refs/heads/2.x from Lars Volker
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=dfb9e16 ]
IMPALA-7006: Add KRPC folders from kudu@334ecafd
cp -a ~/checkout/kudu/src/kudu/{rpc,util,security} be/src/kudu/
Change-Id: I232db2b4ccf5df9aca87b21dea31bfb2735d1ab7
Reviewed-on: http://gerrit.cloudera.org:8080/10757
Reviewed-by: Lars Volker
Tested-by: Lars Volker
> Rebase KRPC onto Kudu upstream repository
> -
>
> Key: IMPALA-7006
> URL: https://issues.apache.org/jira/browse/IMPALA-7006
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Lars Volker
>Priority: Major
> Labels: krpc
> Fix For: Impala 3.1.0
>
>
> We should consider rebasing our KRPC code on top of the latest Kudu upstream
> version. This will keep the two projects more in sync and will allow us to
> make use of recent improvements, e.g. around thread stack collection, without
> having to pick individual changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[
https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810083#comment-16810083
]
ASF subversion and git services commented on IMPALA-7006:
-
Commit b97e0cd555a53057a82dc9c0ad9e0cfe58f3ec66 in impala's branch
refs/heads/2.x from Sailesh Mukil
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=b97e0cd ]
IMPALA-5129: Use Kudu's Kinit code to avoid expensive fork
NOTE: This commit is part of a set of changes for IMPALA-7006. It
contains pieces of a previous commit that need to be cherry picked
again after rebasing the code in be/src/kudu/{util,security,rpc}.
The original commit message is below:
Impala currently kinits by forking off a child process. This
has proved to be expensive in many cases since the subprocess
tries to reserve as much memory as Impala is currently using
which can be quite a lot.
This patch adds a flag called 'use_kudu_kinit' that defaults to
true. When it's true, it uses the Kudu security library's kinit code
that programatically uses the krb5 library to kinit.
When it's false, we run our current path which kicks off the
kinit-thread and forks off a kinit process periodically to reacquire
tickets based on FLAGS_kerberos_reinit_interval.
Converted existing tests in thrift-server-test to run with and
without kerberos. We now run this BE test with kerberos by using
Kudu's MiniKdc utility. This introduces a new dependency on some
kerberos binaries that are checked through FindKerberosPrograms.cmake.
Note that this is only a test dependency and not a dependency for
the impalad binaries and friends. Compilation will still succeed if
the kerberos binaries for the MiniKdc are not found, however, the
thrift-server-test will fail. We run with and without the
'use_kudu_kinit' flag.
TODO: Since the setting up and tearing down of our security code
isn't idempotent, we can run only any one test in a process with
Kerberos now (IMPALA-6085).
Updated bin/bootstrap_system.sh to install new sasl-gssapi
modules and the kerberos binaries required for the MiniKdc.
Also fixed a bug that didn't transfer the environment into 'sudo'
in bin/bootstrap_system.sh.
Testing: Verified with thrift-server-test and also manually on a
live kerberized cluster.
Change-Id: Ie3c6e933c454e7adca69ef03e7d5c0c84b656895
Reviewed-on: http://gerrit.cloudera.org:8080/7938
Reviewed-by: Sailesh Mukil
Tested-by: Impala Public Jenkins
Reviewed-on: http://gerrit.cloudera.org:8080/10763
Reviewed-by: Lars Volker
Tested-by: Lars Volker
> Rebase KRPC onto Kudu upstream repository
> -
>
> Key: IMPALA-7006
> URL: https://issues.apache.org/jira/browse/IMPALA-7006
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Lars Volker
>Priority: Major
> Labels: krpc
> Fix For: Impala 3.1.0
>
>
> We should consider rebasing our KRPC code on top of the latest Kudu upstream
> version. This will keep the two projects more in sync and will allow us to
> make use of recent improvements, e.g. around thread stack collection, without
> having to pick individual changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[
https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810077#comment-16810077
]
ASF subversion and git services commented on IMPALA-7006:
-
Commit 5dbd48f226f1061567da1c381ee2491dab3ceaf4 in impala's branch
refs/heads/2.x from Lars Volker
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=5dbd48f ]
IMPALA-4669: [KUTIL] Add kudu_util library to the build.
NOTE: This commit is part of a set of changes for IMPALA-7006. It
contains pieces of a previous commit that need to be cherry picked
again after rebasing the code in be/src/kudu/{util,security,rpc}.
The original commit message is below:
A few miscellaneous changes to allow kudu_util to compile with Impala.
Add kudu_version.cc to substitute for the version.cc file that is
automatically built during the full Kudu build.
Set LZ4_DISABLE_DEPRECATE_WARNINGS to allow Kudu's compressor utility to
use deprecated names for LZ4 methods.
Add NO_NVM_SUPPORT flag to Kudu build (plan to upstream this later) to
disable building with nvm support, removing a library dependency.
Also remove imported FindOpenSSL.cmake in favour of the standard one provided
by cmake itself.
Finally, a few changes to allow compilation on RHEL5:
* Only use sched_getcpu() if supported
* Only include magic.h if available
* Workaround for kernels that don't have SOCK_NONBLOCK
* Workaround for kernels that don't have O_CLOEXEC (ignore the flag)
* Provide non-working implementation of fallocate()
* Disable inclusion of linux/fiemap.h - although this exists on RHEL5,
it does not compile due to other #includes in env_posix.cc. We disable
the path this is used for, since Impala does not call that code.
* Use Kudu's implementation of pipe(2), preadv(2) and pwritev(2) where
it doesn't exist.
In most cases these changes simply force kutil to revert to a different
implementation that was already written for OSX support - this patch
generalises the logic to provide the implementation whenever the
required function doesn't exist.
This patch compiles on RHEL5.5 and 6.0, SLES11 and 12, Ubuntu 12.04 and
14.04 and Debian 7.0 and 8.0.
Change-Id: I451f02d3e4669e8a548b92fb1445cb2b322659a2
Reviewed-on: http://gerrit.cloudera.org:8080/5715
Tested-by: Impala Public Jenkins
Reviewed-by: Henry Robinson
Reviewed-on: http://gerrit.cloudera.org:8080/10758
Reviewed-by: Michael Ho
Tested-by: Lars Volker
> Rebase KRPC onto Kudu upstream repository
> -
>
> Key: IMPALA-7006
> URL: https://issues.apache.org/jira/browse/IMPALA-7006
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Lars Volker
>Priority: Major
> Labels: krpc
> Fix For: Impala 3.1.0
>
>
> We should consider rebasing our KRPC code on top of the latest Kudu upstream
> version. This will keep the two projects more in sync and will allow us to
> make use of recent improvements, e.g. around thread stack collection, without
> having to pick individual changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[
https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810081#comment-16810081
]
ASF subversion and git services commented on IMPALA-7006:
-
Commit d10b34354c0d1616ed2faf78a6659e9be4aacd66 in impala's branch
refs/heads/2.x from Lars Volker
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=d10b343 ]
IMPALA-4669: [KRPC] Add kudu_rpc library to build
NOTE: This commit is part of a set of changes for IMPALA-7006. It
contains pieces of a previous commit that need to be cherry picked
again after rebasing the code in be/src/kudu/{util,security,rpc}.
The original commit message is below:
Import FindKRPC.cmake from Apache Kudu.
Add some files to protoc-gen-krpc link to allow it to find symbols now
defined within Impala (without linking all of Impala's libraries).
Change-Id: I5693288db90f2e9673b8c88ca4378c3790cba957
Reviewed-on: http://gerrit.cloudera.org:8080/5719
Reviewed-by: Henry Robinson
Tested-by: Impala Public Jenkins
Reviewed-on: http://gerrit.cloudera.org:8080/10760
Reviewed-by: Lars Volker
Tested-by: Lars Volker
> Rebase KRPC onto Kudu upstream repository
> -
>
> Key: IMPALA-7006
> URL: https://issues.apache.org/jira/browse/IMPALA-7006
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Lars Volker
>Priority: Major
> Labels: krpc
> Fix For: Impala 3.1.0
>
>
> We should consider rebasing our KRPC code on top of the latest Kudu upstream
> version. This will keep the two projects more in sync and will allow us to
> make use of recent improvements, e.g. around thread stack collection, without
> having to pick individual changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[ https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542569#comment-16542569 ] ASF subversion and git services commented on IMPALA-7006: - Commit 0459721ccd59deae34548d44c684def4e04b31a6 in impala's branch refs/heads/master from [~lv] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=0459721 ] IMPALA-7006: Pick parts of recent Kudu gutil changes - Include some ASAN macros from gutil (Kudu commit c8724c61) - Pick parts of KUDU-2427 (Kudu commit b7cf3b2e) - Rename constants (Kudu commit e719b5ef) These changes will be subsumed by a proper rebase of GUTIL. Change-Id: Id2dc8c70425e3ac030427ebeb1ec18a44d14d5cb Reviewed-on: http://gerrit.cloudera.org:8080/10769 Tested-by: Impala Public Jenkins Reviewed-by: Lars Volker > Rebase KRPC onto Kudu upstream repository > - > > Key: IMPALA-7006 > URL: https://issues.apache.org/jira/browse/IMPALA-7006 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Lars Volker >Assignee: Lars Volker >Priority: Major > Labels: krpc > Fix For: Impala 3.1.0 > > > We should consider rebasing our KRPC code on top of the latest Kudu upstream > version. This will keep the two projects more in sync and will allow us to > make use of recent improvements, e.g. around thread stack collection, without > having to pick individual changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[
https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542564#comment-16542564
]
ASF subversion and git services commented on IMPALA-7006:
-
Commit ae5eec774d3d064e92b5414ab5db3ed548eb69d6 in impala's branch
refs/heads/master from [~lv]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=ae5eec7 ]
IMPALA-4669: [KRPC] Add kudu_rpc library to build
NOTE: This commit is part of a set of changes for IMPALA-7006. It
contains pieces of a previous commit that need to be cherry picked
again after rebasing the code in be/src/kudu/{util,security,rpc}.
The original commit message is below:
Import FindKRPC.cmake from Apache Kudu.
Add some files to protoc-gen-krpc link to allow it to find symbols now
defined within Impala (without linking all of Impala's libraries).
Change-Id: I5693288db90f2e9673b8c88ca4378c3790cba957
Reviewed-on: http://gerrit.cloudera.org:8080/5719
Reviewed-by: Henry Robinson
Tested-by: Impala Public Jenkins
Reviewed-on: http://gerrit.cloudera.org:8080/10760
Reviewed-by: Lars Volker
Tested-by: Lars Volker
> Rebase KRPC onto Kudu upstream repository
> -
>
> Key: IMPALA-7006
> URL: https://issues.apache.org/jira/browse/IMPALA-7006
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Lars Volker
>Priority: Major
> Labels: krpc
> Fix For: Impala 3.1.0
>
>
> We should consider rebasing our KRPC code on top of the latest Kudu upstream
> version. This will keep the two projects more in sync and will allow us to
> make use of recent improvements, e.g. around thread stack collection, without
> having to pick individual changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[
https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542558#comment-16542558
]
ASF subversion and git services commented on IMPALA-7006:
-
Commit fcf190c4de1fcc291a5356634fd7cd12efa64852 in impala's branch
refs/heads/master from [~lv]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=fcf190c ]
IMPALA-7006: Add KRPC folders from kudu@334ecafd
cp -a ~/checkout/kudu/src/kudu/{rpc,util,security} be/src/kudu/
Change-Id: I232db2b4ccf5df9aca87b21dea31bfb2735d1ab7
Reviewed-on: http://gerrit.cloudera.org:8080/10757
Reviewed-by: Lars Volker
Tested-by: Lars Volker
> Rebase KRPC onto Kudu upstream repository
> -
>
> Key: IMPALA-7006
> URL: https://issues.apache.org/jira/browse/IMPALA-7006
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Lars Volker
>Priority: Major
> Labels: krpc
> Fix For: Impala 3.1.0
>
>
> We should consider rebasing our KRPC code on top of the latest Kudu upstream
> version. This will keep the two projects more in sync and will allow us to
> make use of recent improvements, e.g. around thread stack collection, without
> having to pick individual changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[ https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542561#comment-16542561 ] ASF subversion and git services commented on IMPALA-7006: - Commit 96aa7b477d5172c723e80efa03f747658c16e687 in impala's branch refs/heads/master from [~lv] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=96aa7b4 ] IMPALA-7006: [KSECURITY] Update security library integration This commit is part of a set of changes for IMPALA-7006. It started based on an original change (f51c4435), which integrated Kudu's security folder into our build. This change removes several compile time checks that are now either done in Kudu's own cmake files or that can be removed due to Impala deprecating support for older OS versions in the 3.x line. The removed checks are: HAVE_KRB5_GET_INIT_CREDS_OPT_SET_OUT_CCACHE: We now check for this in Kudu's code. HAVE_KRB5_IS_CONFIG_PRINCIPAL, HAVE_KRB5_GET_INIT_CREDS_OPT_SET_FAST_CCACHE_NAME: These checks are not needed anymore. All OS versions supported by Impala now have sufficiently recent versions of Kerberos. Change-Id: Ifab51d887f5e771ad62eeddc14b9c47f42c3130d Reviewed-on: http://gerrit.cloudera.org:8080/10759 Reviewed-by: Lars Volker Tested-by: Lars Volker > Rebase KRPC onto Kudu upstream repository > - > > Key: IMPALA-7006 > URL: https://issues.apache.org/jira/browse/IMPALA-7006 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Lars Volker >Assignee: Lars Volker >Priority: Major > Labels: krpc > Fix For: Impala 3.1.0 > > > We should consider rebasing our KRPC code on top of the latest Kudu upstream > version. This will keep the two projects more in sync and will allow us to > make use of recent improvements, e.g. around thread stack collection, without > having to pick individual changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[
https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542560#comment-16542560
]
ASF subversion and git services commented on IMPALA-7006:
-
Commit e1d3a616651c93a0164d3e6377f16aaae298d8cd in impala's branch
refs/heads/master from [~lv]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=e1d3a61 ]
IMPALA-4669: [KUTIL] Add kudu_util library to the build.
NOTE: This commit is part of a set of changes for IMPALA-7006. It
contains pieces of a previous commit that need to be cherry picked
again after rebasing the code in be/src/kudu/{util,security,rpc}.
The original commit message is below:
A few miscellaneous changes to allow kudu_util to compile with Impala.
Add kudu_version.cc to substitute for the version.cc file that is
automatically built during the full Kudu build.
Set LZ4_DISABLE_DEPRECATE_WARNINGS to allow Kudu's compressor utility to
use deprecated names for LZ4 methods.
Add NO_NVM_SUPPORT flag to Kudu build (plan to upstream this later) to
disable building with nvm support, removing a library dependency.
Also remove imported FindOpenSSL.cmake in favour of the standard one provided
by cmake itself.
Finally, a few changes to allow compilation on RHEL5:
* Only use sched_getcpu() if supported
* Only include magic.h if available
* Workaround for kernels that don't have SOCK_NONBLOCK
* Workaround for kernels that don't have O_CLOEXEC (ignore the flag)
* Provide non-working implementation of fallocate()
* Disable inclusion of linux/fiemap.h - although this exists on RHEL5,
it does not compile due to other #includes in env_posix.cc. We disable
the path this is used for, since Impala does not call that code.
* Use Kudu's implementation of pipe(2), preadv(2) and pwritev(2) where
it doesn't exist.
In most cases these changes simply force kutil to revert to a different
implementation that was already written for OSX support - this patch
generalises the logic to provide the implementation whenever the
required function doesn't exist.
This patch compiles on RHEL5.5 and 6.0, SLES11 and 12, Ubuntu 12.04 and
14.04 and Debian 7.0 and 8.0.
Change-Id: I451f02d3e4669e8a548b92fb1445cb2b322659a2
Reviewed-on: http://gerrit.cloudera.org:8080/5715
Tested-by: Impala Public Jenkins
Reviewed-by: Henry Robinson
Reviewed-on: http://gerrit.cloudera.org:8080/10758
Reviewed-by: Michael Ho
Tested-by: Lars Volker
> Rebase KRPC onto Kudu upstream repository
> -
>
> Key: IMPALA-7006
> URL: https://issues.apache.org/jira/browse/IMPALA-7006
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Lars Volker
>Priority: Major
> Labels: krpc
> Fix For: Impala 3.1.0
>
>
> We should consider rebasing our KRPC code on top of the latest Kudu upstream
> version. This will keep the two projects more in sync and will allow us to
> make use of recent improvements, e.g. around thread stack collection, without
> having to pick individual changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[ https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542562#comment-16542562 ] ASF subversion and git services commented on IMPALA-7006: - Commit 96aa7b477d5172c723e80efa03f747658c16e687 in impala's branch refs/heads/master from [~lv] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=96aa7b4 ] IMPALA-7006: [KSECURITY] Update security library integration This commit is part of a set of changes for IMPALA-7006. It started based on an original change (f51c4435), which integrated Kudu's security folder into our build. This change removes several compile time checks that are now either done in Kudu's own cmake files or that can be removed due to Impala deprecating support for older OS versions in the 3.x line. The removed checks are: HAVE_KRB5_GET_INIT_CREDS_OPT_SET_OUT_CCACHE: We now check for this in Kudu's code. HAVE_KRB5_IS_CONFIG_PRINCIPAL, HAVE_KRB5_GET_INIT_CREDS_OPT_SET_FAST_CCACHE_NAME: These checks are not needed anymore. All OS versions supported by Impala now have sufficiently recent versions of Kerberos. Change-Id: Ifab51d887f5e771ad62eeddc14b9c47f42c3130d Reviewed-on: http://gerrit.cloudera.org:8080/10759 Reviewed-by: Lars Volker Tested-by: Lars Volker > Rebase KRPC onto Kudu upstream repository > - > > Key: IMPALA-7006 > URL: https://issues.apache.org/jira/browse/IMPALA-7006 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Lars Volker >Assignee: Lars Volker >Priority: Major > Labels: krpc > Fix For: Impala 3.1.0 > > > We should consider rebasing our KRPC code on top of the latest Kudu upstream > version. This will keep the two projects more in sync and will allow us to > make use of recent improvements, e.g. around thread stack collection, without > having to pick individual changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[
https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542568#comment-16542568
]
ASF subversion and git services commented on IMPALA-7006:
-
Commit 1ac9a3f329e3a792c1f3e26272149d05b14732b5 in impala's branch
refs/heads/master from [~joemcdonnell]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=1ac9a3f ]
KUDU-2305: Limit sidecars to INT_MAX and fortify socket code
NOTE: This commit is part of a set of changes for IMPALA-7006. It
contains pieces of a previous commit that need to be cherry picked
again after rebasing the code in be/src/kudu/{util,security,rpc}.
The original commit message is below:
Inspection of the code revealed some other local variables
that could overflow with large messages. This patch takes
two approaches to eliminate the issues.
First, it limits the total size of the messages by limiting
the total size of the sidecars to INT_MAX. The total size
of the protobuf and header components of the message
should be considerably smaller, so limiting the sidecars
to INT_MAX eliminates messages that are larger than UINT_MAX.
This also means that the sidecar offsets, which are unsigned
32-bit integers, are also safe. Given that
FLAGS_rpc_max_message_size is limited to INT_MAX at startup,
the receiver would reject any message this large anyway.
This also helps with the networking codepath, as any given
sidecar will have a size less than INT_MAX, so every Slice
that interacts with Writev() is shorter than INT_MAX.
Second, even with sidecars limited to INT_MAX, the headers
and protobuf parts of the messages mean that certain messages
could still exceed INT_MAX. This patch changes some of the sockets
codepath to tolerate iovec's that reference more than INT_MAX
bytes total. Specifically, it changes Writev()'s nwritten bytes
to an int64_t for both TlsSocket and Socket. TlsSocket works
because it is sending each Slice individually. The first change
limited any given Slice to INT_MAX, so each individual Write()
should not be impacted. For Socket, Writev() uses sendmsg(). It
should do partial network sends to handle this case. Any Write()
call specifies its size with a 32-bit integer, and that will
not be impacted by this patch.
Testing:
- Modified TestRpcSidecarLimits() to verify that sidecars are
limited to INT_MAX bytes.
- Added a test mode to TestRpcSidecarLimits() where it
overrides rpc_max_message_size and sends the maximal
message. This verifies that the client send codepath
can handle the maximal message.
Reviewed-on: http://gerrit.cloudera.org:8080/9601
Reviewed-by: Todd Lipcon
Tested-by: Todd Lipcon
Changes from Kudu version:
- Updated declaration of FLAGS_rpc_max_message_size
in rpc-mgr.cc and added a warning not to set it
larger than INT_MAX.
Change-Id: Id23e518995f2bf2f6bf6b49d5f413f3eaa4e79d1
Reviewed-on: http://gerrit.cloudera.org:8080/9748
Reviewed-by: Michael Ho
Tested-by: Impala Public Jenkins
Reviewed-on: http://gerrit.cloudera.org:8080/10765
Reviewed-by: Lars Volker
Tested-by: Lars Volker
> Rebase KRPC onto Kudu upstream repository
> -
>
> Key: IMPALA-7006
> URL: https://issues.apache.org/jira/browse/IMPALA-7006
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Lars Volker
>Priority: Major
> Labels: krpc
> Fix For: Impala 3.1.0
>
>
> We should consider rebasing our KRPC code on top of the latest Kudu upstream
> version. This will keep the two projects more in sync and will allow us to
> make use of recent improvements, e.g. around thread stack collection, without
> having to pick individual changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7006) Rebase KRPC onto Kudu upstream repository
[
https://issues.apache.org/jira/browse/IMPALA-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542566#comment-16542566
]
ASF subversion and git services commented on IMPALA-7006:
-
Commit bee01825e23fc097c1d8ff58f68afb5141ea57db in impala's branch
refs/heads/master from [~sailesh]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=bee0182 ]
IMPALA-5129: Use Kudu's Kinit code to avoid expensive fork
NOTE: This commit is part of a set of changes for IMPALA-7006. It
contains pieces of a previous commit that need to be cherry picked
again after rebasing the code in be/src/kudu/{util,security,rpc}.
The original commit message is below:
Impala currently kinits by forking off a child process. This
has proved to be expensive in many cases since the subprocess
tries to reserve as much memory as Impala is currently using
which can be quite a lot.
This patch adds a flag called 'use_kudu_kinit' that defaults to
true. When it's true, it uses the Kudu security library's kinit code
that programatically uses the krb5 library to kinit.
When it's false, we run our current path which kicks off the
kinit-thread and forks off a kinit process periodically to reacquire
tickets based on FLAGS_kerberos_reinit_interval.
Converted existing tests in thrift-server-test to run with and
without kerberos. We now run this BE test with kerberos by using
Kudu's MiniKdc utility. This introduces a new dependency on some
kerberos binaries that are checked through FindKerberosPrograms.cmake.
Note that this is only a test dependency and not a dependency for
the impalad binaries and friends. Compilation will still succeed if
the kerberos binaries for the MiniKdc are not found, however, the
thrift-server-test will fail. We run with and without the
'use_kudu_kinit' flag.
TODO: Since the setting up and tearing down of our security code
isn't idempotent, we can run only any one test in a process with
Kerberos now (IMPALA-6085).
Updated bin/bootstrap_system.sh to install new sasl-gssapi
modules and the kerberos binaries required for the MiniKdc.
Also fixed a bug that didn't transfer the environment into 'sudo'
in bin/bootstrap_system.sh.
Testing: Verified with thrift-server-test and also manually on a
live kerberized cluster.
Change-Id: Ie3c6e933c454e7adca69ef03e7d5c0c84b656895
Reviewed-on: http://gerrit.cloudera.org:8080/7938
Reviewed-by: Sailesh Mukil
Tested-by: Impala Public Jenkins
Reviewed-on: http://gerrit.cloudera.org:8080/10763
Reviewed-by: Lars Volker
Tested-by: Lars Volker
> Rebase KRPC onto Kudu upstream repository
> -
>
> Key: IMPALA-7006
> URL: https://issues.apache.org/jira/browse/IMPALA-7006
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Lars Volker
>Priority: Major
> Labels: krpc
> Fix For: Impala 3.1.0
>
>
> We should consider rebasing our KRPC code on top of the latest Kudu upstream
> version. This will keep the two projects more in sync and will allow us to
> make use of recent improvements, e.g. around thread stack collection, without
> having to pick individual changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
