Hi Charles,
This error is Drill's long-winded way of saying you have a direct memory (value
vector) memory leak: you allocated a vector (or buffer) which was never passed
downstream or freed. Looks like quite a few leaked so should be fairly easy to
track down.
On your last batch, did you allocate some vectors that ended up not being used,
maybe?
Thanks,
- Paul
On Sunday, November 11, 2018, 6:49:21 AM PST, Charles Givre
<[email protected]> wrote:
Hi Paul,
Thanks for this. I updated the test and I think I’m close but how I’m getting
different errors. It looks like it isn’t closing the cluster properly, but I’m
not quite sure what to do about this.
— C
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.817 sec <<<
FAILURE! - in org.apache.drill.exec.store.syslog.TestSyslogFormat
org.apache.drill.exec.store.syslog.TestSyslogFormat Time elapsed: 1.817 sec
<<< ERROR!
java.lang.RuntimeException: Exception while closing
at
org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46)
at org.apache.drill.exec.client.DrillClient.close(DrillClient.java:475)
at org.apache.drill.test.ClientFixture.close(ClientFixture.java:242)
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
at org.apache.drill.test.ClusterTest.shutdown(ClusterTest.java:89)
Caused by: java.lang.IllegalStateException: Allocator[ROOT] closed with
outstanding buffers allocated (1).
Allocator(ROOT) 0/2048/6176/4294967296 (res/actual/peak/limit)
child allocators: 0
ledgers: 1
ledger[251] allocator: ROOT), isOwning: true, size: 2048, references: 33,
life: 811185821181057..0, allocatorManager: [177, life: 811185821154333..0]
holds 59 buffers.
DrillBuf[447], udle: [178 1229..1237]
DrillBuf[470], udle: [178 1366..1367]
DrillBuf[457], udle: [178 1282..1290]
DrillBuf[442], udle: [178 1207..1215]
DrillBuf[315], udle: [178 11..1126]
DrillBuf[443], udle: [178 1215..1228]
DrillBuf[465], udle: [178 1350..1351]
DrillBuf[456], udle: [178 1282..1319]
DrillBuf[480], udle: [178 1402..1403]
DrillBuf[452], udle: [178 1262..1270]
DrillBuf[453], udle: [178 1270..1281]
DrillBuf[460], udle: [178 1319..1320]
DrillBuf[473], udle: [178 1375..1380]
DrillBuf[485], udle: [178 1416..1417]
DrillBuf[471], udle: [178 1367..1380]
DrillBuf[463], udle: [178 1328..1350]
DrillBuf[474], udle: [178 1380..1402]
DrillBuf[459], udle: [178 1319..1350]
DrillBuf[461], udle: [178 1320..1350]
DrillBuf[468], udle: [178 1359..1366]
DrillBuf[441], udle: [178 1207..1228]
DrillBuf[488], udle: [178 1425..1426]
DrillBuf[313], udle: [178 0..2048]
DrillBuf[433], udle: [178 1129..1426]
DrillBuf[486], udle: [178 1417..1426]
DrillBuf[436], udle: [178 1130..1206]
DrillBuf[464], udle: [178 1350..1366]
DrillBuf[444], udle: [178 1228..1261]
DrillBuf[466], udle: [178 1351..1366]
DrillBuf[469], udle: [178 1366..1380]
DrillBuf[435], udle: [178 1129..1130]
DrillBuf[449], udle: [178 1261..1281]
DrillBuf[458], udle: [178 1290..1319]
DrillBuf[484], udle: [178 1416..1426]
DrillBuf[454], udle: [178 1281..1319]
DrillBuf[434], udle: [178 1129..1206]
DrillBuf[467], udle: [178 1351..1359]
DrillBuf[317], udle: [178 1129..1426]
DrillBuf[438], udle: [178 1138..1206]
DrillBuf[448], udle: [178 1237..1261]
DrillBuf[462], udle: [178 1320..1328]
DrillBuf[450], udle: [178 1261..1262]
DrillBuf[476], udle: [178 1381..1402]
DrillBuf[478], udle: [178 1389..1402]
DrillBuf[479], udle: [178 1402..1416]
DrillBuf[455], udle: [178 1281..1282]
DrillBuf[475], udle: [178 1380..1381]
DrillBuf[439], udle: [178 1206..1228]
DrillBuf[440], udle: [178 1206..1207]
DrillBuf[477], udle: [178 1381..1389]
DrillBuf[437], udle: [178 1130..1138]
DrillBuf[446], udle: [178 1229..1261]
DrillBuf[451], udle: [178 1262..1281]
DrillBuf[487], udle: [178 1417..1425]
DrillBuf[472], udle: [178 1367..1375]
DrillBuf[482], udle: [178 1403..1411]
DrillBuf[483], udle: [178 1411..1416]
DrillBuf[481], udle: [178 1403..1416]
DrillBuf[445], udle: [178 1228..1229]
reservations: 0
at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:503)
at
org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44)
at org.apache.drill.exec.client.DrillClient.close(DrillClient.java:475)
at org.apache.drill.test.ClientFixture.close(ClientFixture.java:242)
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
at org.apache.drill.test.ClusterTest.shutdown(ClusterTest.java:89)
Results :
Tests in error:
TestSyslogFormat>ClusterTest.shutdown:89 » Runtime Exception while closing
> On Nov 9, 2018, at 16:09, Paul Rogers <[email protected]> wrote:
>
> Hi Charles,
>
> Thanks for the PR. Two suggestions for your test. First, use TupleSchema:
>
> TupleSchema schema = new SchemaBuilder() ... .buildSchema().
>
> BatchSchema has some limitations that TupleSchema overcomes.
>
> Second, when I did a PR that added unions, I normalized the "buildFoo()"
> methods. They are now called .resumeSchema(). Checkout the test
> TestSchemaBuilder for an example.
>
> Just checked, the example in the SchemaBuilder class shows the old names.
> Please file a JIRA ticket for this and I'll submit a PR with the fix.
>
> Thanks,
> - Paul
>
>
>
> On Friday, November 9, 2018, 12:52:56 PM PST, Charles Givre
><[email protected]> wrote:
>
> Hello everyone,
> I just submitted a PR for a format plugin for Syslog formatted data. I’m
> really stuck on writing a unit test however and could use some assistance.
> Basically I’m stuck writing a unit test where the data returns a map. I
> attempted to follow the sample code shown below, however, Java doesn’t seem
> to be finding the build() or buildMap() functions and hence the unit test
> doesn’t work.
>
> <p>
> * Code:<pre><code>
> * BatchSchema batchSchema = new SchemaBuilder()
> * .add("c", MinorType.INT)
> * .addMap("a")
> * .addNullable("b", MinorType.VARCHAR)
> * .add("d", MinorType.INT)
> * .addMap("e") // or .addMapArray("e")
> * .add("f", MinorType.VARCHAR)
> * .buildMap()
> * .add("g", MinorType.INT)
> * .buildMap()
> * .addUnion("h") // or .addList("h")
> * .addType(MinorType.INT)
> * .addMap()
> * .add("h1", MinorType.INT)
> * .buildNested()
> * .addList()
> * .addType(MinorType.BIGINT)
> * .buildNested()
> * .build()
> * .addArray("i", MinorType.BIGINT)
> * .addRepeatedList("j")
> * .addDimension()
> * .addArray(MinorType.VARCHAR)
> * .endDimension()
> * .build()
> * .build();
> * </code</pre>
>
> However, it would not work. The test file can be found here:
> https://github.com/apache/drill/blob/a2365042ee0baa73698f1bb9dd183bb4623bc254/contrib/format-syslog/src/test/java/org/apache/drill/exec/store/syslog/TestSyslogFormat.java
>
> <https://github.com/apache/drill/blob/a2365042ee0baa73698f1bb9dd183bb4623bc254/contrib/format-syslog/src/test/java/org/apache/drill/exec/store/syslog/TestSyslogFormat.java>.
> I commented out the non-working test. Any help would be greatly
> appreciated.
> Thanks!
> —C
>
