Apache Drill Hangout Minutes - 11/1/16
Attendees: Paul, Padma, Sorabh, Boaz, Sudheesh, Vitalii, Roman, Dave O, Arina, Laurent, Kunal, Zelaine I had to leave the hangout at 10:30, so my notes only cover the discussion up till then. 1) Variable width decimal support - Dave O Currently Drill only supports fixed width byte array storage of decimals. Dave has submitted a pull request for DRILL-4834 to add support for storing decimals with variable width byte arrays. Eventually, variable width can replace fixed width, but the pull request doesn't cover that. Dave would like someone in the community to review his pull request. 2) 1.9 release - Sudheesh Sudheesh is collecting pull requests for the release. Some have been reviewed and are waiting to be merged. Sudheesh plans to commit a batch this Wed and another this Friday. He's targeting having a release candidate build available next Monday. Laurent asked about Sudheesh's pull request for DRILL-1950. He asked whether thought had been given to supporting newer Drill clients with older Drill servers. Sudheesh indicated that doing this would entail a breaking change in the protocol, and the plan was to defer doing this for a later release where we may want to make other breaking changes like this.
Re: Apache Drill Hangout Minutes - 11/1/16
Oops, mistake in my notes. For the second item, I meant DRILL-4280, not DRILL-1950. On Tue, Nov 1, 2016 at 3:40 PM, Zelaine Fong wrote: > Attendees: Paul, Padma, Sorabh, Boaz, Sudheesh, Vitalii, Roman, Dave O, > Arina, Laurent, Kunal, Zelaine > > I had to leave the hangout at 10:30, so my notes only cover the discussion > up till then. > > 1) Variable width decimal support - Dave O > > Currently Drill only supports fixed width byte array storage of decimals. > Dave has submitted a pull request for DRILL-4834 to add support for storing > decimals with variable width byte arrays. Eventually, variable width can > replace fixed width, but the pull request doesn't cover that. Dave would > like someone in the community to review his pull request. > > 2) 1.9 release - Sudheesh > > Sudheesh is collecting pull requests for the release. Some have been > reviewed and are waiting to be merged. Sudheesh plans to commit a batch > this Wed and another this Friday. He's targeting having a release > candidate build available next Monday. > > Laurent asked about Sudheesh's pull request for DRILL-1950. He asked > whether thought had been given to supporting newer Drill clients with older > Drill servers. Sudheesh indicated that doing this would entail a breaking > change in the protocol, and the plan was to defer doing this for a later > release where we may want to make other breaking changes like this. >
Re: Apache Drill Hangout Minutes - 11/1/16
Just for clarity, DRILL-4280 is a breaking-protocol change, so is the plan to defer this change to a later release, or to defer bringing back compatibility between newer clients and older servers to a later release? Laurent On Tue, Nov 1, 2016 at 3:43 PM, Zelaine Fong wrote: > Oops, mistake in my notes. For the second item, I meant DRILL-4280, not > DRILL-1950. > > On Tue, Nov 1, 2016 at 3:40 PM, Zelaine Fong wrote: > > > Attendees: Paul, Padma, Sorabh, Boaz, Sudheesh, Vitalii, Roman, Dave O, > > Arina, Laurent, Kunal, Zelaine > > > > I had to leave the hangout at 10:30, so my notes only cover the > discussion > > up till then. > > > > 1) Variable width decimal support - Dave O > > > > Currently Drill only supports fixed width byte array storage of decimals. > > Dave has submitted a pull request for DRILL-4834 to add support for > storing > > decimals with variable width byte arrays. Eventually, variable width can > > replace fixed width, but the pull request doesn't cover that. Dave would > > like someone in the community to review his pull request. > > > > 2) 1.9 release - Sudheesh > > > > Sudheesh is collecting pull requests for the release. Some have been > > reviewed and are waiting to be merged. Sudheesh plans to commit a batch > > this Wed and another this Friday. He's targeting having a release > > candidate build available next Monday. > > > > Laurent asked about Sudheesh's pull request for DRILL-1950. He asked > > whether thought had been given to supporting newer Drill clients with > older > > Drill servers. Sudheesh indicated that doing this would entail a > breaking > > change in the protocol, and the plan was to defer doing this for a later > > release where we may want to make other breaking changes like this. > > >
Re: Apache Drill Hangout Minutes - 11/1/16
Hi Laurent, That's right; this was mentioned in the design document. I am piggybacking on previous changes that break the "newer clients talking to older servers" compatibility. For example, as I understand, some resolved sub-tasks of DRILL-4714 [1] *implicitly* break this compatibility; say the "newer" API that was introduced is used by an application which is talking to an older server. The older server drops the connection, unable to handle the message. In DRILL-4280, there is an *explicit* break in that specific compatibility, and the error message is much cleaner with a version mismatch message. The difference is that the C++ client (unlike the Java client) checks for the server version as well, which make the compatibility break more visible. I am not sure about the plan of action in general about this compatibility. However, I could work around the issue by advertising clients' SASL capability to the server. What do you think? Thank you, Sudheesh [1] https://issues.apache.org/jira/browse/DRILL-4714 On Nov 1, 2016, at 7:49 PM, Laurent Goujon wrote: Just for clarity, DRILL-4280 is a breaking-protocol change, so is the plan to defer this change to a later release, or to defer bringing back compatibility between newer clients and older servers to a later release? Laurent On Tue, Nov 1, 2016 at 3:43 PM, Zelaine Fong wrote: Oops, mistake in my notes. For the second item, I meant DRILL-4280, not DRILL-1950. On Tue, Nov 1, 2016 at 3:40 PM, Zelaine Fong wrote: Attendees: Paul, Padma, Sorabh, Boaz, Sudheesh, Vitalii, Roman, Dave O, Arina, Laurent, Kunal, Zelaine I had to leave the hangout at 10:30, so my notes only cover the discussion up till then. 1) Variable width decimal support - Dave O Currently Drill only supports fixed width byte array storage of decimals. Dave has submitted a pull request for DRILL-4834 to add support for storing decimals with variable width byte arrays. Eventually, variable width can replace fixed width, but the pull request doesn't cover that. Dave would like someone in the community to review his pull request. 2) 1.9 release - Sudheesh Sudheesh is collecting pull requests for the release. Some have been reviewed and are waiting to be merged. Sudheesh plans to commit a batch this Wed and another this Friday. He's targeting having a release candidate build available next Monday. Laurent asked about Sudheesh's pull request for DRILL-1950. He asked whether thought had been given to supporting newer Drill clients with older Drill servers. Sudheesh indicated that doing this would entail a breaking change in the protocol, and the plan was to defer doing this for a later release where we may want to make other breaking changes like this.
Re: Apache Drill Hangout Minutes - 11/1/16
That's a good point that the introduction of new request types for prepared statement and metadata introduced some incompatibility issues, and that's why I added a client/server version exchange so that client can make smart decisions about using these new methods or not (this is the approach used by the ODBC driver, where the new calls will be used only if server advertises 1.9.0 or newer). On Java side, direct users of DrillClient would have to check the version as well, a thing the JDBC driver doesn't. That said, the metadata change is not merged yet (and I will add the extra logic), which is not the case regarding prepared statement support. Also, I verified that the old server doesn't drop the connection: it's actually dropping the request, and not responding to the client :( (best would be to return an error message to say unsupported). Laurent On Tue, Nov 1, 2016 at 8:44 PM, Sudheesh Katkam wrote: > Hi Laurent, > > That's right; this was mentioned in the design document. > > I am piggybacking on previous changes that break the "newer clients talking > to older servers" compatibility. For example, as I understand, some > resolved sub-tasks of DRILL-4714 [1] *implicitly* break this compatibility; > say the "newer" API that was introduced is used by an application which is > talking to an older server. The older server drops the connection, unable > to handle the message. > > In DRILL-4280, there is an *explicit* break in that specific compatibility, > and the error message is much cleaner with a version mismatch message. The > difference is that the C++ client (unlike the Java client) checks for the > server version as well, which make the compatibility break more visible. > > I am not sure about the plan of action in general about this compatibility. > However, I could work around the issue by advertising clients' SASL > capability to the server. What do you think? > > Thank you, > Sudheesh > > [1] https://issues.apache.org/jira/browse/DRILL-4714 > > On Nov 1, 2016, at 7:49 PM, Laurent Goujon wrote: > > Just for clarity, DRILL-4280 is a breaking-protocol change, so is the plan > to defer this change to a later release, or to defer bringing back > compatibility between newer clients and older servers to a later release? > > Laurent > > On Tue, Nov 1, 2016 at 3:43 PM, Zelaine Fong wrote: > > Oops, mistake in my notes. For the second item, I meant DRILL-4280, not > DRILL-1950. > > On Tue, Nov 1, 2016 at 3:40 PM, Zelaine Fong wrote: > > Attendees: Paul, Padma, Sorabh, Boaz, Sudheesh, Vitalii, Roman, Dave O, > Arina, Laurent, Kunal, Zelaine > > I had to leave the hangout at 10:30, so my notes only cover the > > discussion > > up till then. > > 1) Variable width decimal support - Dave O > > Currently Drill only supports fixed width byte array storage of decimals. > Dave has submitted a pull request for DRILL-4834 to add support for > > storing > > decimals with variable width byte arrays. Eventually, variable width can > replace fixed width, but the pull request doesn't cover that. Dave would > like someone in the community to review his pull request. > > 2) 1.9 release - Sudheesh > > Sudheesh is collecting pull requests for the release. Some have been > reviewed and are waiting to be merged. Sudheesh plans to commit a batch > this Wed and another this Friday. He's targeting having a release > candidate build available next Monday. > > Laurent asked about Sudheesh's pull request for DRILL-1950. He asked > whether thought had been given to supporting newer Drill clients with > > older > > Drill servers. Sudheesh indicated that doing this would entail a > > breaking > > change in the protocol, and the plan was to defer doing this for a later > release where we may want to make other breaking changes like this. >
Re: Apache Drill Hangout Minutes - 11/1/16
(+dev@drill.apache.org bcc u...@drill.apache.org) other options: - let the user decide between plain auth and sasl auth - let the client could send a flag to announce support for SASL auth, and do not send the credentials in the first handshake. An old server would then send AUTH_FAILED (and drop the connection) but the client would retry with the plain credentials, whereas the new server would send AUTH_REQUIRED as expected. Laurent Laurent On Wed, Nov 2, 2016 at 3:30 PM, Sudheesh Katkam wrote: > I am going to update the pull request so that both will be "ok". > > This implies that username/password credentials will be sent to the server > twice, during handshake and during SASL exchange. And sending credentials > through handshake will be deprecated (and removed in a future release). > > Thank you, > Sudheesh > > On Wed, Nov 2, 2016 at 2:58 PM, Jacques Nadeau wrote: > > > Since I'm not that close to DRILL-4280, I wanted to clarify expectation: > > > > > > <1.9 Client <==> 1.9 Server (ok) > > 1.9 Client <==> <1.9 Server (fails) > > > > Is that correct? > > > > > > > > > > > > > > -- > > Jacques Nadeau > > CTO and Co-Founder, Dremio > > > > On Tue, Nov 1, 2016 at 8:44 PM, Sudheesh Katkam > > wrote: > > > > > Hi Laurent, > > > > > > That's right; this was mentioned in the design document. > > > > > > I am piggybacking on previous changes that break the "newer clients > > talking > > > to older servers" compatibility. For example, as I understand, some > > > resolved sub-tasks of DRILL-4714 [1] *implicitly* break this > > compatibility; > > > say the "newer" API that was introduced is used by an application which > > is > > > talking to an older server. The older server drops the connection, > unable > > > to handle the message. > > > > > > In DRILL-4280, there is an *explicit* break in that specific > > compatibility, > > > and the error message is much cleaner with a version mismatch message. > > The > > > difference is that the C++ client (unlike the Java client) checks for > the > > > server version as well, which make the compatibility break more > visible. > > > > > > I am not sure about the plan of action in general about this > > compatibility. > > > However, I could work around the issue by advertising clients' SASL > > > capability to the server. What do you think? > > > > > > Thank you, > > > Sudheesh > > > > > > [1] https://issues.apache.org/jira/browse/DRILL-4714 > > > > > > On Nov 1, 2016, at 7:49 PM, Laurent Goujon wrote: > > > > > > Just for clarity, DRILL-4280 is a breaking-protocol change, so is the > > plan > > > to defer this change to a later release, or to defer bringing back > > > compatibility between newer clients and older servers to a later > release? > > > > > > Laurent > > > > > > On Tue, Nov 1, 2016 at 3:43 PM, Zelaine Fong > wrote: > > > > > > Oops, mistake in my notes. For the second item, I meant DRILL-4280, > not > > > DRILL-1950. > > > > > > On Tue, Nov 1, 2016 at 3:40 PM, Zelaine Fong > wrote: > > > > > > Attendees: Paul, Padma, Sorabh, Boaz, Sudheesh, Vitalii, Roman, Dave O, > > > Arina, Laurent, Kunal, Zelaine > > > > > > I had to leave the hangout at 10:30, so my notes only cover the > > > > > > discussion > > > > > > up till then. > > > > > > 1) Variable width decimal support - Dave O > > > > > > Currently Drill only supports fixed width byte array storage of > decimals. > > > Dave has submitted a pull request for DRILL-4834 to add support for > > > > > > storing > > > > > > decimals with variable width byte arrays. Eventually, variable width > can > > > replace fixed width, but the pull request doesn't cover that. Dave > would > > > like someone in the community to review his pull request. > > > > > > 2) 1.9 release - Sudheesh > > > > > > Sudheesh is collecting pull requests for the release. Some have been > > > reviewed and are waiting to be merged. Sudheesh plans to commit a > batch > > > this Wed and another this Friday. He's targeting having a release > > > candidate build available next Monday. > > > > > > Laurent asked about Sudheesh's pull request for DRILL-1950. He asked > > > whether thought had been given to supporting newer Drill clients with > > > > > > older > > > > > > Drill servers. Sudheesh indicated that doing this would entail a > > > > > > breaking > > > > > > change in the protocol, and the plan was to defer doing this for a > later > > > release where we may want to make other breaking changes like this. > > > > > >