Re: 1.4.1 release

[James Peach]

I think MESOS-8169 is a candidate, but I don't be able to get to it until next 
week


> On Nov 3, 2017, at 1:48 AM, Qian Zhang  wrote:
> 
> And I will backport MESOS-8051 to 1.2.x, 1.3.x and 1.4.x.
> 
> 
> Regards,
> Qian Zhang
> 
> On Fri, Nov 3, 2017 at 9:01 AM, Qian Zhang  wrote:
> We want to backport https://reviews.apache.org/r/62518/ to 1.2.x, 1.3.x and 
> 1.4.x, James will work on it.
> 
> 
> Regards,
> Qian Zhang
> 
> On Fri, Nov 3, 2017 at 12:11 AM, Kapil Arya  wrote:
> Please reply to this email if you have pending patches to be backported to 
> 1.4.x as we are aiming to cut a release candidate for 1.4.1 early next week.
> 
> Thanks,
> Anand and Kapil
> 
> 

Re: DC/OS (Mesos) portability

[Joseph Wu]

It isn't clear to me how DC/OS would benefit from (ongoing) work to
create/push Mesos packages.  DC/OS downloads and builds all of its
component parts from source.

Also, we (Mesos devs) are hoping to get more frameworks to move away from
using libmesos (including the API shims), in favor of using the HTTP APIs
instead.  So we have a dis-incentive to provide a libmesos bundle.

On Fri, Nov 3, 2017 at 8:23 AM, Tomas Barton  wrote:

> Hi,
>
> I'd like to contribute to DC/OS with a Debian/Suse/... support.
> Surprisingly on Debian most of the compatibility issues could be solved by
> a sequence of symlinks.
>
> Why Mesos dev list? :)
>
> Currently the biggest issue is connected to distributing libmesos-bundle
> tar archive, which contain the libmesos.so library and several others. The
> library is dynamically linked with certain libcurl,  libssl, libsvn etc.
> that might differ between distributions.
>
> I can think of a few solutions:
>  1. Compile Mesos (master and agent) using static build (which as I
> understood aren't currently fully supported/propagated).
>  2. Generate bundle during automatic builds for certain supported
> distributions.
>  3. Include libmesos in standard distribution channels - rpm, deb packages
> (that might take same time).
>
> The last solution would be the best, but Mesos release cycle is very
> different from distributions release cycle. It might be complicated to
> synchronize.
>
> I coudn't find scripts for generating libmesos-bundle, but it's a archive
> with libraries from build server, e.g.
> https://downloads.mesosphere.io/libmesos-bundle/libmesos-
> bundle-1.10-1.4-63e0814.tar.gz
> (32MB).
>
> So the question is, whether Mesos website could provide prebuild libmesos
> bundle for each release and platform, that could be afterwards used e.g. in
> DC/OS packages?
>
> Last issue might be connected to an executor that eventually might need OS
> family ENV variable with OS release version, so that it can fetch
> corresponding libbundle archive. Such information is typically parsed from
> `uname -a` or `lsb_release -sri` (if available). This way DC/OS could be
> running on a cluster with diverse OS versions/distributions.
>
> Thanks for your time! I'd like to hear your opinion.
>
> Regards,
> Tomas Barton
>

Re: Binary RPM packages for CentOS

[Vinod Kone]

This is great to see! Thanks for driving this Kapil.

Once the official packages are vetted, we should update our website to
point users to it. For new users, getting started with binary packages is
infinitely better than having to build from source.

Folks, please help us by testing this packages and provide feedback.

On Fri, Nov 3, 2017 at 10:32 AM, Kapil Arya  wrote:

> Hi All,
>
> We have some updates regarding the binary RPM build/release process for
> Mesos.
>
> Here are some important bullet points:
>
> * Jira: https://issues.apache.org/jira/browse/MESOS-7981
> * The RPM packages are now built using a jenkins job [1] on the Apache CI.
> * The build scripts are part of the Mesos source tree (support/packaging).
> * Packages are distributed using bintray [2].
> * Currently, we are using the mesos org under bintray. Once we have more
>   confidence in the packages, we'll move under the apache org to benefit
> from
>   the superior resource limits.
>
> Here are the steps to install the CentOS 7 packages from bintray:
>
> *$* cat > /tmp/bintray-mesos-el.repo < #bintray-mesos-el - packages by mesos from Bintray
> [bintray-mesos-el]
> name=bintray-mesos-el
> baseurl=https://dl.bintray.com/mesos/el/7/x86_64
> gpgcheck=0
> repo_gpgcheck=0
> enabled=1
> EOF
>
> *$* sudo mv /tmp/bintray-mesos-el.repo
> /etc/yum.repos.d/bintray-mesos-el.repo
>
> *$* sudo yum update
>
> *$* sudo yum install mesos
>
>
> To start, we have released 1.4.0 packages in the hope to get some feedback
> from
> the community before releasing rpms for other supported version.
>
> Future work:
> * Release RC packages in bintray.com/mesos/el-testing repo.
> * Release (rotating) nightly builds in bintray.com/mesos/el-unstable repo.
> * Create/release debian/ubuntu packages with the help from community
> members who
>   are already publishing such packages.
>
> Best,
> Kapil
>
> [1]: https://builds.apache.org/job/Mesos/job/Packaging/job/CentosRPMs
> [2]: https://bintray.com/mesos/
>

Concerning master's validation of ExecutorID upon slave reregistration

[James DeFelice]

I've filed a JIRA for it here:
https://issues.apache.org/jira/browse/MESOS-8169

The change I'm proposing would change validation (for slave reregistration)
to only enforce ExecutorID uniqueness in the context of (FrameworkID,
SlaveID) as opposed to simply SlaveID as currently implemented.

This change might cause trouble for any frameworks that expect ExecutorID
to be unique across every non-terminal executor for every framework running
on a given slave. That said, ExecutorID is a framework-generated ID and so
it's uniqueness *should* be scoped to a framework to begin with.

Please comment on the linked JIRA if you're concerned about the proposed
change.

Thanks,
James

[GitHub] mesos issue #248: Fix master validation, incorrect detection of dup Executor...

[abudnik]

Github user abudnik commented on the issue:

https://github.com/apache/mesos/pull/248
  
If some framework relies on the assumption that there shouldn't be multiple 
executors running with the same executor_id on the same agent, then this change 
might lead to bugs.
It would be great to post an email into dev/user-lists regarding this 
change.


---

[GitHub] mesos issue #248: Fix master validation, incorrect detection of dup Executor...

[jdef]

Github user jdef commented on the issue:

https://github.com/apache/mesos/pull/248
  
@abudnik the master.hpp code you cited is already scoped to Framework. I 
don't think we need to change anything there.


---

[GitHub] mesos issue #248: Fix master validation, incorrect detection of dup Executor...

[abudnik]

Github user abudnik commented on the issue:

https://github.com/apache/mesos/pull/248
  
Should we also update comment 
https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L78-L82 ?


---

[GitHub] mesos issue #248: Fix master validation, incorrect detection of dup Executor...

[abudnik]

Github user abudnik commented on the issue:

https://github.com/apache/mesos/pull/248
  
We should also update here 
https://github.com/apache/mesos/blob/master/src/master/master.hpp#L2665-L2667
and maybe in other places, where master/agent checks for duplicate executor 
ids.


---

Binary RPM packages for CentOS

[Kapil Arya]

Hi All,

We have some updates regarding the binary RPM build/release process for
Mesos.

Here are some important bullet points:

* Jira: https://issues.apache.org/jira/browse/MESOS-7981
* The RPM packages are now built using a jenkins job [1] on the Apache CI.
* The build scripts are part of the Mesos source tree (support/packaging).
* Packages are distributed using bintray [2].
* Currently, we are using the mesos org under bintray. Once we have more
  confidence in the packages, we'll move under the apache org to benefit
from
  the superior resource limits.

Here are the steps to install the CentOS 7 packages from bintray:

*$* cat > /tmp/bintray-mesos-el.repo 

[GitHub] mesos pull request #248: Fix master validation, incorrect detection of dup E...

[jpeach]

Github user jpeach commented on a diff in the pull request:

https://github.com/apache/mesos/pull/248#discussion_r148843748
  
--- Diff: src/master/validation.cpp ---
@@ -298,7 +299,9 @@ Option reregisterSlave(
 const vector& frameworkInfos)
 {
   hashset frameworkIDs;
-  hashset executorIDs;
+
+  typedef pair IDPair;
+  hashset executorIDs;
--- End diff --

Mesos doesn't use convenience typedefs.
```C
hashset> executorIDs
```

In subsequent lines, I believe that this is ok:
```
auto id = std::make_pair(executor.framework_id(), executor.executor_id());
```


---

[GitHub] mesos pull request #248: Fix master validation, incorrect detection of dup E...

[jpeach]

Github user jpeach commented on a diff in the pull request:

https://github.com/apache/mesos/pull/248#discussion_r148844153
  
--- Diff: src/master/validation.cpp ---
@@ -350,12 +353,13 @@ Option reregisterSlave(
 }
 
 if (executor.has_executor_id()) {
-  if (executorIDs.contains(executor.executor_id())) {
+  IDPair idpair = IDPair(executor.framework_id(), 
executor.executor_id());
+  if (executorIDs.contains(idpair)) {
 return Error("Executor has a duplicate ExecutorID '" +
  stringify(executor.executor_id()) + "'");
--- End diff --

We should update this error message:
```
Error("Framework '" + executor.framework_id() +
  '" has a duplicate ExecutorID '" +
  stringify(executor.executor_id()) + "'");
```


---

Re: mesos health checks

[Alex Rukletsov]

+ dev list for visibility and history.

Okay, let's dig into this a little bit : ).

First, it is true that Marathon and Mesos HTTP health checks are not
equivalent. It's not just 1xx status codes, you can't have multiple Mesos
health checks for example. I don't understand why you say that the operator
should know that failed is an expected response. It is not! Health checks
do not have a concept of "not ready yet", grace period serves this purpose.
The health check has failed because the contract had been violated: 111 is
considered a failure. If you think that 1xx codes should be treated as
success — let's have this discussion separately, probably on the dev list
(btw, k8s does the same

).

Second, are you sure about the status code in the second case? The error
does not say anything about empty body, but empty reply. From what I can see
,
(52) means a misbehaving server. If you're convinced that your server
returned a proper HTTP response with some status code but with empty body,
please file a bug report against Mesos jira.

On Fri, Nov 3, 2017 at 2:20 PM, Alex Rukletsov  wrote:

> Tomas, can I reply to you and cc devlist to have our discussion logged
> publicly?
>
>
> On Fri, Nov 3, 2017 at 10:43 AM, Tomas Barton 
> wrote:
>
>> Hi Alex,
>>
>> I'm quite ok with the current contract, treat "codes between 200 and 399
>> as success" seems reasonable for me. We're using code < 200 for "not
>> ready yet" and >= 500 for error states.
>>
>> But that's not really the problem. While Marathon's implementation only
>> checked the HTTP code, curl tends to be too smart. Meaning that going from
>> Marathon healthcheck to MESOS based might introduce some incompatibility.
>>
>> For example:
>>
>> (2017-11-02 19:31:25) [INFO] Request: 127.0.0.1:44172 0x1fcc44f0
>> HTTP/1.1 GET /health
>> (2017-11-02 19:31:25) [INFO] Response: 0x1fcc44f0 /health 111 0
>> I1102 19:31:25.548070 23822 checker_process.cpp:959] HTTP health check
>> for task 'reql-dev.3c83761f-c004-11e7-acb9-be622fe0971d' returned: 111
>> W1102 19:31:25.548195 23822 health_checker.cpp:317] HTTP health check for
>> task 'reql-dev.3c83761f-c004-11e7-acb9-be622fe0971d' failed: Unexpected
>> HTTP response code: 111
>>
>> This is sort of ok, the operator should know that "failed: Unexpected
>> HTTP response code: 111" isn't really a failure but an expected response.
>>
>> But in order to get this we had to hack into HTTP server and introduce
>> some "special" HTTP codes.
>>
>> Another component where health checks on Marathon we responding as
>> expected, behaves funny with MESOS_HTTP:
>>
>> W1102 10:50:38.637907 6 health_checker.cpp:307] HTTP health check for
>> task 'xxx' failed: curl exited with status 52: curl: (52) Empty reply from
>> server
>> I1102 10:50:38.637949 6 health_checker.cpp:333] Ignoring failure of
>> HTTP health check for task 'xxx': still in grace period
>>
>> In this case the response code was either 100 or 111. Hard to tell from
>> the logs as the return code is not logged. The problem is, that the
>> component is written in Java, where some library for creating simple
>> webserver responds to /health endpoint is using underneath pretty standard
>> Jetty server. And Jetty decided that responses with code 1xx doesn't have
>> to send body response. On the other side curl thinks that HTTP response
>> with 1xx should have body response, thus the error code (52) Empty reply
>> from server. Maybe we should simply respond with HTTP 418 I'm a teapot,
>> meaning that the tea is not ready yet :)
>>
>> So, the question is, could be curl configured in a way where it doesn't
>> check for body content? And if body is present include it in logs?
>>
>> Or should I file bug reports to all web servers to include Mesos
>> compatible http responses? :)
>>
>> Thanks!
>> Tomas
>>
>>
>> On 2 November 2017 at 19:58, Alex Rukletsov  wrote:
>>
>>> Hi Tomas!
>>>
>>> I wanted to make health checks as simple as possible. I had looked at
>>> what aws, k8s, and nomad do and decided that I will not support
>>> customization for return codes unless someone shows me a very good reason
>>> to do so. Such customization is not easy, once you start it, people will
>>> want more and more, think about API: enumerate "good" codes, enumerate
>>> "bad" codes, specify "good" range, specify "bad" range, specify set of
>>> "good" ranges, and so on.
>>>
>>> Regarding the empty reply, why an empty reply should be considered ok?
>>> The contract is very explicit: "Default executors treat return codes
>>> between 200 and 399 as success; custom executors may employ a different
>>> strategy, e.g. leveraging the `statuses` field."
>>>
>>> And it actually should affect app scaling, as the task should be
>>> considered unhealthy.
>>>
>>> So—give me a good reason to chan

Mesos 1.2.3 (the last 1.2.x release)

[Adam Bordelon]

Please reply to this email if you have pending patches to backport to 1.2.x.
I'm aiming to cut a 1.2.3 next week. This will be the last 1.2.x release.

I see 3 JIRAs targeted for 1.2.3, but none of them look quite ready to land:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20MESOS%20AND%20%22Target%20Version%2Fs%22%20%3D%201.2.3%20AND%20status%20!%3D%20Resolved
If they aren't in the 1.2.x branch by next week, I'll drop the 1.2.3 target
version from them.

DC/OS (Mesos) portability

[Tomas Barton]

Hi,

I'd like to contribute to DC/OS with a Debian/Suse/... support.
Surprisingly on Debian most of the compatibility issues could be solved by
a sequence of symlinks.

Why Mesos dev list? :)

Currently the biggest issue is connected to distributing libmesos-bundle
tar archive, which contain the libmesos.so library and several others. The
library is dynamically linked with certain libcurl,  libssl, libsvn etc.
that might differ between distributions.

I can think of a few solutions:
 1. Compile Mesos (master and agent) using static build (which as I
understood aren't currently fully supported/propagated).
 2. Generate bundle during automatic builds for certain supported
distributions.
 3. Include libmesos in standard distribution channels - rpm, deb packages
(that might take same time).

The last solution would be the best, but Mesos release cycle is very
different from distributions release cycle. It might be complicated to
synchronize.

I coudn't find scripts for generating libmesos-bundle, but it's a archive
with libraries from build server, e.g.
https://downloads.mesosphere.io/libmesos-bundle/libmesos-bundle-1.10-1.4-63e0814.tar.gz
(32MB).

So the question is, whether Mesos website could provide prebuild libmesos
bundle for each release and platform, that could be afterwards used e.g. in
DC/OS packages?

Last issue might be connected to an executor that eventually might need OS
family ENV variable with OS release version, so that it can fetch
corresponding libbundle archive. Such information is typically parsed from
`uname -a` or `lsb_release -sri` (if available). This way DC/OS could be
running on a cluster with diverse OS versions/distributions.

Thanks for your time! I'd like to hear your opinion.

Regards,
Tomas Barton

[GitHub] mesos issue #248: Fix master validation, incorrect detection of dup Executor...

[jdef]

Github user jdef commented on the issue:

https://github.com/apache/mesos/pull/248
  
https://issues.apache.org/jira/browse/MESOS-8169


---

[GitHub] mesos issue #248: Fix master validation, incorrect detection of dup Executor...

[jpeach]

Github user jpeach commented on the issue:

https://github.com/apache/mesos/pull/248
  
@jdef Good find! Can you please make a JIRA for this?


---

[GitHub] mesos issue #248: Fix master validation, incorrect detection of dup Executor...

[jdef]

Github user jdef commented on the issue:

https://github.com/apache/mesos/pull/248
  
@jpeach it looks like you were in here recently, what are your thoughts on 
this?


---

[GitHub] mesos pull request #248: Fix master validation, incorrect detection of dup E...

[jdef]

Github user jdef commented on a diff in the pull request:

https://github.com/apache/mesos/pull/248#discussion_r148790566
  
--- Diff: src/master/validation.cpp ---
@@ -350,12 +353,13 @@ Option reregisterSlave(
 }
 
 if (executor.has_executor_id()) {
-  if (executorIDs.contains(executor.executor_id())) {
+  IDPair fep = IDPair(executor.framework_id(), executor.executor_id());
--- End diff --

already got feedback that `fep` is probably not a good choice for a name 
here


---

Re: Sandbox life cycle /age

[Venkat Morampudi]

Hi Benjamin,

Apologies for the delay. GC seem be working fine. Folders older than 2 hours 
are being deleted. After change of config and restart, Mesos agent took some 
time to delete the old folder that are around before the restart. I may have 
jumped the  gun.

Thanks,
Venkat

> On Oct 30, 2017, at 1:01 PM, Benjamin Mahler  wrote:
> 
> Hi Venkat,
> 
> You're seeing that files with a modification time greater than your gc
> delay of 2 hours are *not* getting deleted? Can you show a full
> listing of /var/lib/mesos/slave/slaves/?
> Is there more than 1 entry there?
> 
> On Fri, Oct 27, 2017 at 8:43 AM, Venkat Morampudi  
>> wrote:
> 
>> Hi Tomek,
>> 
>> After changing GC delay to 2hrs, the existing sandbox folders that are
>> older than the “Max allowed age” are not deleted. Here are the logs
>> 
>> Logs entire before and after the change:
>> 
>> I1027 15:00:48.055465 12861 slave.cpp:4615] Current disk usage 21.63%. Max
>> allowed age: 1.367499658088657days
>> I1027 15:02:37.720451 30693 slave.cpp:4615] Current disk usage 21.60%. Max
>> allowed age: 1.368035520611667hrs
>> 
>> Executor info from the node:
>> 
>> 
>> [techops@kaiju-dcos-privateslave27 ~]$ date
>> Fri Oct 27 15:41:59 UTC 2017
>> [techops@kaiju-dcos-privateslave27 ~]$ ls -l /var/lib/mesos/slave/slaves/
>> 3fd7ffe9-945b-4ae1-968e-2c397585960c-S201/frameworks/35e600c2-6
>> f43-402c-856f-9084c0040187-002/executors/
>> total 452
>> drwxr-xr-x. 3 root root 4096 Oct 26 09:43 74861.0.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 09:56 74861.10.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 10:31 74867.0.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 11:07 74871.3.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 11:45 74875.10.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 13:43 74886.0.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 13:45 74886.2.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 13:51 74886.8.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 13:56 74886.9.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 13:59 74887.1.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 14:42 74895.0.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 14:58 74899.7.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 14:59 74900.8.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 15:12 74902.0.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 17:05 74938.0.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 17:10 74940.0.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 17:11 74942.0.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 17:30 74944.3.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 17:45 74951.0.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 17:45 74951.1.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 18:07 74959.7.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 18:47 74971.0.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 19:18 74981.3.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 20:14 74992.0.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 20:28 74995.3.0
>> drwxr-xr-x. 3 root root 4096 Oct 26 20:59 75001.1.0
>> 
>> 
>> Thanks,
>> Venkat
>> 
>>> On Oct 27, 2017, at 6:42 AM, Tomek Janiszewski 
>> wrote:
>>> 
>>> Low GC delay menas files will be deleted more often. I don't' think there
>>> will be any performance problem but low GC means you will lose your
>>> sandboxes earlier and they are useful for debugging purposes.
>>> 
>>> pt., 27 paź 2017 o 04:40 użytkownik Venkat Morampudi <
>>> venkatmoramp...@gmail.com  
>>> >> 
>>> napisał:
>>> 
 Hi Tomek,
 
 Thanks for the quick reply. After digging a bit into Mesos code we were
 able understand that age actually mean threshold age. Anything older
>> than
 the “age" would be GCed. We are going to try different setting starting
 with "--gc_disk_headroom=.2 --gc_delay=2hrs”. Is there any downside of
>> the
 going with very low GC delay?
 
 Thanks,
 Venkat
 
 
> On Oct 26, 2017, at 4:28 PM, Tomek Janiszewski 
 wrote:
> 
>> gc_delay * max(0.0, (1.0 - gc_disk_headroom - disk usage))
> 
> *Example:*
> gc_delay = 7days
> gc_disk_headroom = 0.1
> disk_usage = 0.8
> 7 * max(0.0, 1 - 0.1 - 0.8) = 7 * max(0.0, 0.1) = 0.7 days = 16 h 48
>> min
> 
> Can you show some logs containging information about GC?
> 
> pt., 27 paź 2017 o 00:43 użytkownik Venkat Morampudi <
> venkatmoramp...@gmail.com  
> > 
> > venkatmoramp...@gmail.com >> napisał:
> 
>> Hello,
>> In our production env, we noticed that our disk filled up because one
>> framework had a lot of failed/completed executors folders laying
>> around.
>> The folders eventually filled up the disk.
>> 
>> 
>> 228M
>> 
 /mnt/resource/slaves/c8674097-6e67-4609-b022-3e11de380fe5-
>> S2/frameworks/35e600c2-6f43-402c-856f-9084c0040187-002/executors/52334.1.0
>> 228M
>> 
 /mnt/resource/slaves/c8674097-6e67-4609-b022

[GitHub] mesos pull request #248: Fix master validation, incorrect detection of dup E...

[jdef]

GitHub user jdef opened a pull request:

https://github.com/apache/mesos/pull/248

Fix master validation, incorrect detection of dup ExecutorID.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jdef/mesos 
jdef_master_validation_reregisterslave

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/mesos/pull/248.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #248


commit 27127e7731a40baedcc88af6929525c9f8b1e5e7
Author: James DeFelice 
Date:   2017-11-03T13:37:18Z

Fix master validation, incorrect detection of dup ExecutorID.




---

Re: 1.4.1 release

[Qian Zhang]

And I will backport MESOS-8051 to 1.2.x, 1.3.x and 1.4.x.


Regards,
Qian Zhang

On Fri, Nov 3, 2017 at 9:01 AM, Qian Zhang  wrote:

> We want to backport https://reviews.apache.org/r/62518/ to 1.2.x, 1.3.x
> and 1.4.x, James will work on it.
>
>
> Regards,
> Qian Zhang
>
> On Fri, Nov 3, 2017 at 12:11 AM, Kapil Arya  wrote:
>
>> Please reply to this email if you have pending patches to be backported
>> to 1.4.x as we are aiming to cut a release candidate for 1.4.1 early next
>> week.
>>
>> Thanks,
>> Anand and Kapil
>>
>
>