[GitHub] aravi5 commented on a change in pull request #1626: DRILL-6855: Query from non-existent proxy user fails with "No default schema selected" when impersonation is enabled

[GitBox]

aravi5 commented on a change in pull request #1626: DRILL-6855: Query from 
non-existent proxy user fails with "No default schema selected" when 
impersonation is enabled
URL: https://github.com/apache/drill/pull/1626#discussion_r255288516
 
 

 ##
 File path: 
exec/java-exec/src/main/java/org/apache/calcite/jdbc/DynamicRootSchema.java
 ##
 @@ -115,8 +116,25 @@ public void loadSchemaFactory(String schemaName, boolean 
caseSensitive) {
   schemaPlus.add(wrapper.getName(), wrapper);
 }
   }
-} catch(ExecutionSetupException | IOException ex) {
+} catch(ExecutionSetupException ex) {
   logger.warn("Failed to load schema for \"" + schemaName + "\"!", ex);
+} catch (IOException iex) {
+  // We can't proceed further without a schema, throw a runtime exception.
+  UserException.Builder exceptBuilder =
+  UserException
+  .resourceError(iex)
+  .message("Failed to create schema tree.")
+  .addContext("IOException: ", iex.getMessage());
+
+  // Improve the error message for client side.
+  final String errorMsg = "Error getting user info for current user";
+  if (iex.getMessage().startsWith(errorMsg)) {
+final String contextString = "[Hint: Username is absent in connection 
URL or doesn't " +
+ "exist on Drillbit node. Please 
specify a username in connection " +
+ "URL which is present on Drillbit 
node.]";
+exceptBuilder.addContext(contextString);
+  }
+  throw exceptBuilder.build(logger);
 
 Review comment:
   I agree with your point that `ExecutionSetupException` should be handled 
similar to `IOException`. I did not make the changes since I did not fully 
implications and was trying to limit the impact. May be @arina-ielchiieva / 
@chunhui-shi could weigh in their thoughts here.
   
   I have added a utility that could provide hints, if any. This way we can 
have a single place for all hints.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (DRILL-6914) Query with RuntimeFilter and SemiJoin fails with IllegalStateException: Memory was leaked by query

[Boaz Ben-Zvi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boaz Ben-Zvi resolved DRILL-6914.
-
Resolution: Fixed

The interaction between the Hash-Join spill and the runtime filter was fixed in 
PR #1622. Testing with the latest code works OK (no memory leaks).

 

> Query with RuntimeFilter and SemiJoin fails with IllegalStateException: 
> Memory was leaked by query
> --
>
> Key: DRILL-6914
> URL: https://issues.apache.org/jira/browse/DRILL-6914
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Abhishek Ravi
>Assignee: Boaz Ben-Zvi
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: 23cc1af3-0e8e-b2c9-a889-a96504988d6c.sys.drill, 
> 23cc1b7c-5b5c-d123-5e72-6d7d2719df39.sys.drill
>
>
> Following query fails on TPC-H SF 100 dataset when 
> exec.hashjoin.enable.runtime_filter = true AND planner.enable_semijoin = true.
> Note that the query does not fail if any one of them or both are disabled.
> {code:sql}
> set `exec.hashjoin.enable.runtime_filter` = true;
> set `exec.hashjoin.runtime_filter.max.waiting.time` = 1;
> set `planner.enable_broadcast_join` = false;
> set `planner.enable_semijoin` = true;
> select
>  count(*) as row_count
> from
>  lineitem l1
> where
>  l1.l_shipdate IN (
>  select
>  distinct(cast(l2.l_shipdate as date))
>  from
>  lineitem l2);
> reset `exec.hashjoin.enable.runtime_filter`;
> reset `exec.hashjoin.runtime_filter.max.waiting.time`;
> reset `planner.enable_broadcast_join`;
> reset `planner.enable_semijoin`;
> {code}
>  
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. 
> Memory leaked: (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567 
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010] 
> (state=,code=0)
> java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked 
> by query. Memory leaked: (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567 
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> at 
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:536)
> at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:640)
> at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
> at 
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
> at sqlline.BufferedRows.(BufferedRows.java:37)
> at sqlline.SqlLine.print(SqlLine.java:1716)
> at sqlline.Commands.execute(Commands.java:949)
> at sqlline.Commands.sql(Commands.java:882)
> at sqlline.SqlLine.dispatch(SqlLine.java:725)
> at sqlline.SqlLine.runCommands(SqlLine.java:1779)
> at sqlline.Commands.run(Commands.java:1485)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
> at sqlline.SqlLine.dispatch(SqlLine.java:722)
> at sqlline.SqlLine.initArgs(SqlLine.java:458)
> at sqlline.SqlLine.begin(SqlLine.java:514)
> at sqlline.SqlLine.start(SqlLine.java:264)
> at sqlline.SqlLine.main(SqlLine.java:195)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM 
> ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: 
> (134217728)
> Allocator(frag:1:0) 800/134217728/172453568/70126322567 
> (res/actual/peak/limit)
> Fragment 1:0
> Please, refer to logs for more information.
> [Error Id: ccee18b3-c3ff-4fdb-b314-23a6cfed0a0e on qa-node185.qa.lab:31010]
> at 
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
> at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at 
> 

[jira] [Created] (DRILL-7034) Window function over a malformed CSV file crashes the JVM

[Boaz Ben-Zvi (JIRA)]

Boaz Ben-Zvi created DRILL-7034:
---

 Summary: Window function over a malformed CSV file crashes the JVM 
 Key: DRILL-7034
 URL: https://issues.apache.org/jira/browse/DRILL-7034
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Relational Operators
Affects Versions: 1.15.0
Reporter: Boaz Ben-Zvi


The JVM crashes executing window functions over (an ordered) CSV file with a 
small format issue - an empty line.

To create: Take the following simple `a.csvh` file:
{noformat}
amount
10
11
{noformat}

And execute a simple window function like
{code:sql}
select max(amount) over(order by amount) FROM dfs.`/data/a.csvh`;
{code}

Then add an empty line between the `10` and the `11`:
{noformat}
amount
10

11
{noformat}

 and try again:
{noformat}
0: jdbc:drill:zk=local> select max(amount) over(order by amount) FROM 
dfs.`/data/a.csvh`;
+-+
| EXPR$0  |
+-+
| 10  |
| 11  |
+-+
2 rows selected (3.554 seconds)
0: jdbc:drill:zk=local> select max(amount) over(order by amount) FROM 
dfs.`/data/a.csvh`;
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0001064aeae7, pid=23450, tid=0x6103
#
# JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build 
1.8.0_181-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode bsd-amd64 
compressed oops)
# Problematic frame:
# J 6719% C2 
org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.memcmp(JIIJII)I (188 
bytes) @ 0x0001064aeae7 [0x0001064ae920+0x1c7]
#
# Core dump written. Default location: /cores/core or core.23450
#
# An error report file with more information is saved as:
# /Users/boazben-zvi/IdeaProjects/drill/hs_err_pid23450.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Abort trap: 6 (core dumped)
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] vdiravka commented on a change in pull request #1627: DRILL-7014: Format plugin for LTSV files

[GitBox]

vdiravka commented on a change in pull request #1627: DRILL-7014: Format plugin 
for LTSV files
URL: https://github.com/apache/drill/pull/1627#discussion_r255199611
 
 

 ##
 File path: 
contrib/format-ltsv/src/main/java/org/apache/drill/exec/store/ltsv/LTSVFormatPlugin.java
 ##
 @@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.ltsv;
+
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.proto.UserBitShared;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.RecordReader;
+import org.apache.drill.exec.store.RecordWriter;
+import org.apache.drill.exec.store.dfs.DrillFileSystem;
+import org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin;
+import org.apache.drill.exec.store.dfs.easy.EasyWriter;
+import org.apache.drill.exec.store.dfs.easy.FileWork;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.util.List;
+
+public class LTSVFormatPlugin extends EasyFormatPlugin 
{
+
+private static final boolean IS_COMPRESSIBLE = false;
+private static final String DEFAULT_NAME = "ltsv";
+private LTSVFormatPluginConfig config;
+
+private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(LTSVFormatPlugin.class);
+
+public LTSVFormatPlugin(String name, DrillbitContext context, 
Configuration fsConf, StoragePluginConfig storageConfig) {
+this(name, context, fsConf, storageConfig, new 
LTSVFormatPluginConfig());
+}
+
+public LTSVFormatPlugin(String name, DrillbitContext context, 
Configuration fsConf, StoragePluginConfig config, LTSVFormatPluginConfig 
formatPluginConfig) {
+super(name, context, fsConf, config, formatPluginConfig, true, false, 
false, IS_COMPRESSIBLE, formatPluginConfig.getExtensions(), DEFAULT_NAME);
+this.config = formatPluginConfig;
+}
+
+@Override
+public RecordReader getRecordReader(FragmentContext context, 
DrillFileSystem dfs, FileWork fileWork,
+List columns, String 
userName) throws ExecutionSetupException {
+return new LTSVRecordReader(context, fileWork.getPath(), dfs, columns, 
config);
+}
+
+
+@Override
+public int getReaderOperatorType() {
+// TODO Is it correct??
+return UserBitShared.CoreOperatorType.JSON_SUB_SCAN_VALUE;
 
 Review comment:
   @shimamoto @cgivre 
   The good news:
   by simply rebasing to the master branch you can see the Protobuf job in 
Travis.
   If it fails, it means your changes require Protobuf regenerating. 
   So you can do protobuf regenerating or just copy the `git diff` from the 
Protobuf Travis job and apply it as a patch in your IDE. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Re: Travis CI improvements

[Kunal Khatua]

Very useful, Vova! Thanks.

Does the protobuf regeneration run always or is it conditional to detecting 
changes in the protobuf files? It might be worth testing for changes to .proto 
files before attempting to regenerate the Java and C++ protobuf files. 
Otherwise, you'd be increasing the effective runtime on Travis by 50% all 
across the commits (incl those unrelated to protobuf).

~ Kunal
On 2/8/2019 3:36:06 AM, Vova Vysotskyi  wrote:
Hi all,

Recently there are more PRs which require changes in protobuf files, but
sometimes contributors may forget to regenerate them. It was the reason for
creating DRILL-7031 .

Before the fix for this Jira, there was a single job which builds Drill,
checks licenses and runs unit tests.
In the fix for this Jira, this job was splitted into two jobs: the first
one only runs unit tests and the second one builds Drill, checks the
licenses and regenerates both Java and C++ protobuf files. For the case, if
after regeneration changes are found, job will fail.

So time required for finishing Travis job is reduced to 29 minutes (time of
the longest job), but total time for both jobs exceeds current (was 32
mins, but now 29+15 mins). Current build bay be found here:
https://travis-ci.org/apache/drill/builds/490483425?utm_source=github_status_medium=notification

Except for the check for changes in protobuf files, in the case of failure
will be printed diff for changed classes which may be copied and applied as
a patch. Build with the failed protobuf check may be found here:
https://travis-ci.org/vdiravka/drill/jobs/490016316.

Kind regards,
Volodymyr Vysotskyi

[GitHub] kkhatua commented on issue #1608: DRILL-6960: AutoLimit the size of ResultSet for a WebUI (or REST) client

[GitBox]

kkhatua commented on issue #1608: DRILL-6960: AutoLimit the size of ResultSet 
for a WebUI (or REST) client
URL: https://github.com/apache/drill/pull/1608#issuecomment-461893766
 
 
   @ihuzenko / @vvysotskyi 
   Please review the squashed commits. 
   It includes changes requested by you and the PR summary is mentioned in the 
comment: https://github.com/apache/drill/pull/1608#issuecomment-461889067
   Screenshots and test queries are in the following comments


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kkhatua commented on issue #1608: DRILL-6960: AutoLimit the size of ResultSet for a WebUI (or REST) client

[GitBox]

kkhatua commented on issue #1608: DRILL-6960: AutoLimit the size of ResultSet 
for a WebUI (or REST) client
URL: https://github.com/apache/drill/pull/1608#issuecomment-461893168
 
 
   Tested for the following queries with Limit = 13
   
   ```sql
   --ShowDatabases [Expected: NoLimit on SQL]
   show databases
   
   --SqlKind: WITH [Expected: Limit applied to SQL]
   with X1 (r_regionkey, r_name, n_nationkey, n_name) as 
 (select r.r_regionkey as r_regionkey, r.r_name as r_name, n.n_nationkey as 
n_nationkey, n.n_name as n_name from dfs.par100.region r, dfs.par100.nation n 
where r.r_regionkey=n.n_regionkey)
select X1.n_name as nation, X1.r_name as region, count(*) as tally 
 from X1, dfs.par100.customer c 
 where X1.n_nationkey = c.c_nationkey
 group by X1.n_name, X1.r_name
 order by tally
   
   --SqlKind: ORDER_BY [Expected: Limit applied to SQL]
   select r.r_name as region, n.n_name as nation
, count(*) as tally 
 from dfs.par100.region r, dfs.par100.nation n 
 where r.r_regionkey=n.n_regionkey
 group by n.n_name, r.r_name
 order by tally
   
   -- SqlKind: UNION outputting 30 rows [Expected: Limit applied to SQL]
   select r_name as name, r_regionkey as code from dfs.par100.region
   UNION
   select n_name as name, n_nationkey as code from dfs.par100.nation 
   
   -- Simple Join outputting 25 rows [Expected: Limit applied to SQL]
   select r.r_regionkey as r_regionkey, r.r_name as r_name, n.n_nationkey as 
n_nationkey, n.n_name as n_name from dfs.par100.region r, dfs.par100.nation n 
where r.r_regionkey=n.n_regionkey
   
   -- Huge Join outputting 600M rows [Expected: Limit applied to SQL and 4-node 
execution in <2 minutes]
   select * from dfs.par100.lineitem l, dfs.par100.orders o where o.o_orderkey 
= l.l_orderkey;
   
   -- Simple Join with existing Limit 22 [Expected: Limit 13 applied to SQL]
   select r.r_regionkey as r_regionkey, r.r_name as r_name, n.n_nationkey as 
n_nationkey, n.n_name as n_name from dfs.par100.region r, dfs.par100.nation n 
where r.r_regionkey=n.n_regionkey limit 22
   
   -- Simple Join with existing Limit 11 [Expected: NoLimit on SQL]
   select r.r_regionkey as r_regionkey, r.r_name as r_name, n.n_nationkey as 
n_nationkey, n.n_name as n_name from dfs.par100.region r, dfs.par100.nation n 
where r.r_regionkey=n.n_regionkey limit 11
   
   --CreateTable [Expected: NoLimit on SQL]
   create temporary table dummy ( nation )  as 
   select n_name from dfs.par100.nation
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kkhatua commented on issue #1608: DRILL-6960: AutoLimit the size of ResultSet for a WebUI (or REST) client

[GitBox]

kkhatua commented on issue #1608: DRILL-6960: AutoLimit the size of ResultSet 
for a WebUI (or REST) client
URL: https://github.com/apache/drill/pull/1608#issuecomment-461892922
 
 
   **Screenshot of Error Handling:**
   
![image](https://user-images.githubusercontent.com/4335237/52496526-6b76b880-2b88-11e9-8ae7-a28a4784d96c.png)
   
   
   **Screenshot of results for the query**
   ```sql
   -- Simple Join with existing Limit 22 [Expected: Limit 13 applied to SQL]
   select r.r_regionkey as r_regionkey, r.r_name as r_name, n.n_nationkey as 
n_nationkey, n.n_name as n_name 
   from dfs.par100.region r, dfs.par100.nation n 
   where r.r_regionkey=n.n_regionkey 
   limit 22
   ```
   
![image](https://user-images.githubusercontent.com/4335237/52496646-c6101480-2b88-11e9-8df7-fdc012ba9c4b.png)
   
   
   **Screenshot of profile with Limit applied (overriding the existing larger 
limit 22)**
   
![image](https://user-images.githubusercontent.com/4335237/52496795-374fc780-2b89-11e9-8f20-702f590fe861.png)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] kkhatua commented on issue #1608: DRILL-6960: AutoLimit the size of ResultSet for a WebUI (or REST) client

[GitBox]

kkhatua commented on issue #1608: DRILL-6960: AutoLimit the size of ResultSet 
for a WebUI (or REST) client
URL: https://github.com/apache/drill/pull/1608#issuecomment-461889067
 
 
   **DRILL-6960: AutoLimit the size of ResultSet for a WebUI (or REST) client**
   1. Check if username (if impersonation is enabled) and limits are valid 
inputs
   2. Switch `help` from onclick to a hovering tooltip
   3. Protobuf Changes for QueryProfile to indicate if a query was run with a 
'Limit wrap'
 a. UserBitShared.proto
 b. Java files (after # 3.a)
 c. C++ files (after # 3.a)
   4. Server is able to detect the submitted autoLimit and apply the wrapper 
around the input SQL. For this, it evaluates SqlNode.getKind() and only 
wrapping with limit if it is a QUERY. If autoLimit is not applicable, remove 
the value to avoid incorrectly marking the profile during execution.
   5. WebUI will render a pinned warning indicating that the result set size 
was limited. This should help explain two identical looking queries with 
different runtimes, because on of them ran with an implicit Limit Wrap
   6. Queries can be resubmitted from their profile page, for which the 
optional limit is automatically enabled set to the value for the original 
profile. 
   (That means a user does not have to remember to enable the limit wrap with 
the same value again, and can focus on the query instead)
   7. The results page indicates if the number of records being rendered was 
auto-limited


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Drill Resource Management Project

[Sorabh Hamirwasia]

Hi All,
Please find below the link[1] to the slides for Drill Resource Management
project which Hanu, Karthik and I presented on this week's hangout. For
details you can refer to design document at [2]. The implementation details
section of design document is still WIP. All the work for this project will
be tracked using DRILL-7026 [3]

[1]:
https://docs.google.com/presentation/d/1tGCyfckUlp19cO2DvUKd6aEEOFp1-Z6QVc7BCL670vc/edit?usp=sharing
[2]:
https://docs.google.com/document/d/1yUfaFJvejeC3lJoY3Y778tQCGJLca356sHQln84zyw0/edit?usp=sharing
[3]: https://issues.apache.org/jira/browse/DRILL-7026

Thanks,
Sorabh

[jira] [Created] (DRILL-7033) Java heap out of memory issue when trying to query huge data

[Gayathri (JIRA)]

Gayathri created DRILL-7033:
---

 Summary: Java heap out of memory issue when trying to query huge 
data 
 Key: DRILL-7033
 URL: https://issues.apache.org/jira/browse/DRILL-7033
 Project: Apache Drill
  Issue Type: Bug
Reporter: Gayathri


We are currently working on Apache drill for querying JSON objects. We are not 
using any dfs plugin to store the json data. All the JSON data is sent to query 
directly using convert_from('','json') function. If the data which is passed is 
more than 1 MB, then we are getting "Java Heap Out of Memory issue". When 
trying to monitor drill through jmc, we found that the old gen GC is occupying 
more than 7GB Heap. (Heap configured is 8GB)

Could you please provide any inputs on this?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Travis CI improvements

[Arina Yelchiyeva]

Great improvements. Thanks, Vova!

Kind regards,
Arina

> On Feb 8, 2019, at 1:35 PM, Vova Vysotskyi  wrote:
> 
> Hi all,
> 
> Recently there are more PRs which require changes in protobuf files, but
> sometimes contributors may forget to regenerate them. It was the reason for
> creating DRILL-7031 .
> 
> Before the fix for this Jira, there was a single job which builds Drill,
> checks licenses and runs unit tests.
> In the fix for this Jira, this job was splitted into two jobs: the first
> one only runs unit tests and the second one builds Drill, checks the
> licenses and regenerates both Java and C++ protobuf files. For the case, if
> after regeneration changes are found, job will fail.
> 
> So time required for finishing Travis job is reduced to 29 minutes (time of
> the longest job), but total time for both jobs exceeds current (was 32
> mins, but now 29+15 mins). Current build bay be found here:
> https://travis-ci.org/apache/drill/builds/490483425?utm_source=github_status_medium=notification
> 
> Except for the check for changes in protobuf files, in the case of failure
> will be printed diff for changed classes which may be copied and applied as
> a patch. Build with the failed protobuf check may be found here:
> https://travis-ci.org/vdiravka/drill/jobs/490016316.
> 
> Kind regards,
> Volodymyr Vysotskyi

Travis CI improvements

[Vova Vysotskyi]

Hi all,

Recently there are more PRs which require changes in protobuf files, but
sometimes contributors may forget to regenerate them. It was the reason for
creating DRILL-7031 .

Before the fix for this Jira, there was a single job which builds Drill,
checks licenses and runs unit tests.
In the fix for this Jira, this job was splitted into two jobs: the first
one only runs unit tests and the second one builds Drill, checks the
licenses and regenerates both Java and C++ protobuf files. For the case, if
after regeneration changes are found, job will fail.

So time required for finishing Travis job is reduced to 29 minutes (time of
the longest job), but total time for both jobs exceeds current (was 32
mins, but now 29+15 mins). Current build bay be found here:
https://travis-ci.org/apache/drill/builds/490483425?utm_source=github_status_medium=notification

Except for the check for changes in protobuf files, in the case of failure
will be printed diff for changed classes which may be copied and applied as
a patch. Build with the failed protobuf check may be found here:
https://travis-ci.org/vdiravka/drill/jobs/490016316.

Kind regards,
Volodymyr Vysotskyi

[GitHub] asfgit closed pull request #1633: DRILL-7024: Refactor ColumnWriter to simplify type-conversion shim

[GitBox]

asfgit closed pull request #1633: DRILL-7024: Refactor ColumnWriter to simplify 
type-conversion shim
URL: https://github.com/apache/drill/pull/1633
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] asfgit closed pull request #1636: DRILL-7031: Add Travis job that runs protobuf generation command and checks if all protobufs are up-to-date

[GitBox]

asfgit closed pull request #1636: DRILL-7031: Add Travis job that runs protobuf 
generation command and checks if all protobufs are up-to-date
URL: https://github.com/apache/drill/pull/1636
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] asfgit closed pull request #1632: DRILL-6780: Caching dependencies for CircleCI

[GitBox]

asfgit closed pull request #1632: DRILL-6780: Caching dependencies for CircleCI
URL: https://github.com/apache/drill/pull/1632
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services