Re: EVF Log Regex Errors
Hi Charles,
Turns out that there are two problems here. First, I mucked up the Jackson
serialization of the schema objects. Second, you need to use the Joda format
(with "HH") as we discussed. Once both those changes are made, things seem to
work (at least in unit tests.)
There is a PR for the fix. Please review.
Thanks,
- Paul
On Tuesday, July 16, 2019, 6:45:01 PM PDT, Charles Givre
wrote:
Hi Paul,
Thanks for the response. Unfortunately, I tried simply setting a fieldName and
got an error.
"ssdlog": {
"type": "logRegex",
"regex":
"(\\w{3}\\s\\d{1,2}\\s\\d{4}\\s\\d{2}:\\d{2}:\\d{2})\\s+(\\w+)\\[(\\d+)\\]:\\s(.*?(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}).*?)",
"extension": "ssdlog",
"maxErrors": 10,
"schema": [{"fieldName": "test"}]
},
--C
> On Jul 16, 2019, at 7:08 PM, Paul Rogers wrote:
>
> Hi Charles,
>
> Thanks much for the feedback. I'll take a look.
>
> A quick look at your config suggests that the timestamp might be the issue.
> As I recall, there were no such tests in the unit test class. So, perhaps
> something slipped through. (We should add a test for this case.)
>
>
> In EVF, we use the Joda (not Java 8) date/time classes. [1] (We do this for
> obscure reasons related to how Drill handles intervals, and the fact that the
> Java 8 date/time classes are not a full replacement for Joda.)
>
> With Joda, your format should be: "MMM dd HH:mm:ss" (Note the upper case
> "H"). Try this to see if it gets you unstuck.
>
> What we should really do is support SQL format strings. These are not
> standard, but the Postgres format seem common [2]. Someone added this feature
> to Drill a while back, so we must have a Postgres-to-Joda format converter in
> the code somewhere we could use.
>
> Thanks,
> - Paul
>
>
> [1]
> https://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
>
> [2] https://www.postgresql.org/docs/9.1/functions-formatting.html
>
>
>
>
> On Tuesday, July 16, 2019, 02:23:50 PM PDT, Charles Givre
> wrote:
>
>
> Hello All,
> First, a big thank you Paul for updating the log regex reader to the new EVF
> framework. I am having a little trouble getting it to work however...
> Here is my config:
>
> ,
> "ssdlog": {
> "type": "logRegex",
> "regex":
>"(\\w{3}\\s\\d{1,2}\\s\\d{4}\\s\\d{2}:\\d{2}:\\d{2})\\s+(\\w+)\\[(\\d+)\\]:\\s(.*?(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}).*?)",
> "extension": "ssdlog",
> "maxErrors": 10,
> "schema": [
> {"fieldName":"eventDate"}
> ]
> },
>
> This works if I leave the schema null, however if I attempt to populate it, I
> get JSON errors. This was what I originally had:
>
> "schema" : [ {
> "fieldName" : "eventDate",
> "fieldType" : "TIMESTAMP",
> "format" : "MMM dd hh:mm:ss"
> }, {
> "fieldName" : "process_name"
> }, {
> "fieldName" : "pid",
> "fieldType" : "INT"
> }, {
> "fieldName" : "message"
> }, {
> "fieldName" : "src_ip"
> } ]
>
> which worked.
>
>
> Also, I am working on updating a few format plugins and kept getting the
> following error when I try to run unit tests:
>
> at org.apache.drill.test.ClusterFixture.(ClusterFixture.java:152)
> at
>org.apache.drill.test.ClusterFixtureBuilder.build(ClusterFixtureBuilder.java:283)
> at org.apache.drill.test.ClusterTest.startCluster(ClusterTest.java:83)
> at
>org.apache.drill.exec.store.excel.TestExcelFormat.setup(TestExcelFormat.java:49)
> Caused by: com.typesafe.config.ConfigException$Missing: No configuration
> setting found for key 'drill.exec.grace_period_ms'
> at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:136)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155)
> at
>com.typesafe.config.impl.SimpleConfig.getConfigNumber(SimpleConfig.java:170)
> at com.typesafe.config.impl.SimpleConfig.getInt(SimpleConfig.java:181)
> at org.apache.drill.common.config.NestedConfig.getInt(NestedConfig.java:96)
> at org.apache.drill.common.config.DrillConfig.getInt(DrillConfig.java:44)
> at org.apache.drill.common.config.NestedConfig.getInt(NestedConfig.java:96)
> at org.apache.drill.common.config.DrillConfig.getInt(DrillConfig.java:44)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:160)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:138)
> at
>org.apache.drill.test.ClusterFixture.startDrillbits(ClusterFixture.java:228)
> at org.apache.drill.test.ClusterFixture.(ClusterFixture.java:146)
> ... 3 more
>
>
> Process finished with exit code 255
>
> I understand t
Re: EVF Log Regex Errors
Hi Charles,
Please file a JIRA ticket and include a short file and the config. I'll take a
look in a few days and figure out what's broken.
Thanks,
- Paul
On Tuesday, July 16, 2019, 6:45:01 PM PDT, Charles Givre
wrote:
Hi Paul,
Thanks for the response. Unfortunately, I tried simply setting a fieldName and
got an error.
"ssdlog": {
"type": "logRegex",
"regex":
"(\\w{3}\\s\\d{1,2}\\s\\d{4}\\s\\d{2}:\\d{2}:\\d{2})\\s+(\\w+)\\[(\\d+)\\]:\\s(.*?(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}).*?)",
"extension": "ssdlog",
"maxErrors": 10,
"schema": [{"fieldName": "test"}]
},
--C
> On Jul 16, 2019, at 7:08 PM, Paul Rogers wrote:
>
> Hi Charles,
>
> Thanks much for the feedback. I'll take a look.
>
> A quick look at your config suggests that the timestamp might be the issue.
> As I recall, there were no such tests in the unit test class. So, perhaps
> something slipped through. (We should add a test for this case.)
>
>
> In EVF, we use the Joda (not Java 8) date/time classes. [1] (We do this for
> obscure reasons related to how Drill handles intervals, and the fact that the
> Java 8 date/time classes are not a full replacement for Joda.)
>
> With Joda, your format should be: "MMM dd HH:mm:ss" (Note the upper case
> "H"). Try this to see if it gets you unstuck.
>
> What we should really do is support SQL format strings. These are not
> standard, but the Postgres format seem common [2]. Someone added this feature
> to Drill a while back, so we must have a Postgres-to-Joda format converter in
> the code somewhere we could use.
>
> Thanks,
> - Paul
>
>
> [1]
> https://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
>
> [2] https://www.postgresql.org/docs/9.1/functions-formatting.html
>
>
>
>
> On Tuesday, July 16, 2019, 02:23:50 PM PDT, Charles Givre
> wrote:
>
>
> Hello All,
> First, a big thank you Paul for updating the log regex reader to the new EVF
> framework. I am having a little trouble getting it to work however...
> Here is my config:
>
> ,
> "ssdlog": {
> "type": "logRegex",
> "regex":
>"(\\w{3}\\s\\d{1,2}\\s\\d{4}\\s\\d{2}:\\d{2}:\\d{2})\\s+(\\w+)\\[(\\d+)\\]:\\s(.*?(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}).*?)",
> "extension": "ssdlog",
> "maxErrors": 10,
> "schema": [
> {"fieldName":"eventDate"}
> ]
> },
>
> This works if I leave the schema null, however if I attempt to populate it, I
> get JSON errors. This was what I originally had:
>
> "schema" : [ {
> "fieldName" : "eventDate",
> "fieldType" : "TIMESTAMP",
> "format" : "MMM dd hh:mm:ss"
> }, {
> "fieldName" : "process_name"
> }, {
> "fieldName" : "pid",
> "fieldType" : "INT"
> }, {
> "fieldName" : "message"
> }, {
> "fieldName" : "src_ip"
> } ]
>
> which worked.
>
>
> Also, I am working on updating a few format plugins and kept getting the
> following error when I try to run unit tests:
>
> at org.apache.drill.test.ClusterFixture.(ClusterFixture.java:152)
> at
>org.apache.drill.test.ClusterFixtureBuilder.build(ClusterFixtureBuilder.java:283)
> at org.apache.drill.test.ClusterTest.startCluster(ClusterTest.java:83)
> at
>org.apache.drill.exec.store.excel.TestExcelFormat.setup(TestExcelFormat.java:49)
> Caused by: com.typesafe.config.ConfigException$Missing: No configuration
> setting found for key 'drill.exec.grace_period_ms'
> at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:136)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155)
> at
>com.typesafe.config.impl.SimpleConfig.getConfigNumber(SimpleConfig.java:170)
> at com.typesafe.config.impl.SimpleConfig.getInt(SimpleConfig.java:181)
> at org.apache.drill.common.config.NestedConfig.getInt(NestedConfig.java:96)
> at org.apache.drill.common.config.DrillConfig.getInt(DrillConfig.java:44)
> at org.apache.drill.common.config.NestedConfig.getInt(NestedConfig.java:96)
> at org.apache.drill.common.config.DrillConfig.getInt(DrillConfig.java:44)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:160)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:138)
> at
>org.apache.drill.test.ClusterFixture.startDrillbits(ClusterFixture.java:228)
> at org.apache.drill.test.ClusterFixture.(ClusterFixture.java:146)
> ... 3 more
>
>
> Process finished with exit code 255
>
> I understand that I have to set the variable drill.exec.grace_period_ms, but
> I'm not sure how/where to do this. Here is the beginning of my unit test
> code:
>
> @ClassRule
> public stat
Re: EVF Log Regex Errors
Hi Paul,
Thanks for the response. Unfortunately, I tried simply setting a fieldName and
got an error.
"ssdlog": {
"type": "logRegex",
"regex":
"(\\w{3}\\s\\d{1,2}\\s\\d{4}\\s\\d{2}:\\d{2}:\\d{2})\\s+(\\w+)\\[(\\d+)\\]:\\s(.*?(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}).*?)",
"extension": "ssdlog",
"maxErrors": 10,
"schema": [{"fieldName": "test"}]
},
--C
> On Jul 16, 2019, at 7:08 PM, Paul Rogers wrote:
>
> Hi Charles,
>
> Thanks much for the feedback. I'll take a look.
>
> A quick look at your config suggests that the timestamp might be the issue.
> As I recall, there were no such tests in the unit test class. So, perhaps
> something slipped through. (We should add a test for this case.)
>
>
> In EVF, we use the Joda (not Java 8) date/time classes. [1] (We do this for
> obscure reasons related to how Drill handles intervals, and the fact that the
> Java 8 date/time classes are not a full replacement for Joda.)
>
> With Joda, your format should be: "MMM dd HH:mm:ss" (Note the upper case
> "H"). Try this to see if it gets you unstuck.
>
> What we should really do is support SQL format strings. These are not
> standard, but the Postgres format seem common [2]. Someone added this feature
> to Drill a while back, so we must have a Postgres-to-Joda format converter in
> the code somewhere we could use.
>
> Thanks,
> - Paul
>
>
> [1]
> https://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
>
> [2] https://www.postgresql.org/docs/9.1/functions-formatting.html
>
>
>
>
>On Tuesday, July 16, 2019, 02:23:50 PM PDT, Charles Givre
> wrote:
>
>
> Hello All,
> First, a big thank you Paul for updating the log regex reader to the new EVF
> framework. I am having a little trouble getting it to work however...
> Here is my config:
>
> ,
> "ssdlog": {
> "type": "logRegex",
> "regex":
> "(\\w{3}\\s\\d{1,2}\\s\\d{4}\\s\\d{2}:\\d{2}:\\d{2})\\s+(\\w+)\\[(\\d+)\\]:\\s(.*?(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}).*?)",
> "extension": "ssdlog",
> "maxErrors": 10,
> "schema": [
> {"fieldName":"eventDate"}
> ]
> },
>
> This works if I leave the schema null, however if I attempt to populate it, I
> get JSON errors. This was what I originally had:
>
> "schema" : [ {
> "fieldName" : "eventDate",
> "fieldType" : "TIMESTAMP",
> "format" : "MMM dd hh:mm:ss"
> }, {
> "fieldName" : "process_name"
> }, {
> "fieldName" : "pid",
> "fieldType" : "INT"
> }, {
> "fieldName" : "message"
> }, {
> "fieldName" : "src_ip"
> } ]
>
> which worked.
>
>
> Also, I am working on updating a few format plugins and kept getting the
> following error when I try to run unit tests:
>
> at org.apache.drill.test.ClusterFixture.(ClusterFixture.java:152)
> at
> org.apache.drill.test.ClusterFixtureBuilder.build(ClusterFixtureBuilder.java:283)
> at org.apache.drill.test.ClusterTest.startCluster(ClusterTest.java:83)
> at
> org.apache.drill.exec.store.excel.TestExcelFormat.setup(TestExcelFormat.java:49)
> Caused by: com.typesafe.config.ConfigException$Missing: No configuration
> setting found for key 'drill.exec.grace_period_ms'
> at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:136)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155)
> at
> com.typesafe.config.impl.SimpleConfig.getConfigNumber(SimpleConfig.java:170)
> at com.typesafe.config.impl.SimpleConfig.getInt(SimpleConfig.java:181)
> at
> org.apache.drill.common.config.NestedConfig.getInt(NestedConfig.java:96)
> at org.apache.drill.common.config.DrillConfig.getInt(DrillConfig.java:44)
> at
> org.apache.drill.common.config.NestedConfig.getInt(NestedConfig.java:96)
> at org.apache.drill.common.config.DrillConfig.getInt(DrillConfig.java:44)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:160)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:138)
> at
> org.apache.drill.test.ClusterFixture.startDrillbits(ClusterFixture.java:228)
> at org.apache.drill.test.ClusterFixture.(ClusterFixture.java:146)
> ... 3 more
>
>
> Process finished with exit code 255
>
> I understand that I have to set the variable drill.exec.grace_period_ms, but
> I'm not sure how/where to do this. Here is the beginning of my unit test
> code:
>
> @ClassRule
> public static final BaseDirTestWatcher dirTestWatcher = new
> BaseDirTestWatcher();
>
> @BeforeClass
> public static void setup() throws Exception {
>
> ClusterTest.startCluster(ClusterFixt
Re: EVF Log Regex Errors
Hi Charles,
Thanks much for the feedback. I'll take a look.
A quick look at your config suggests that the timestamp might be the issue. As
I recall, there were no such tests in the unit test class. So, perhaps
something slipped through. (We should add a test for this case.)
In EVF, we use the Joda (not Java 8) date/time classes. [1] (We do this for
obscure reasons related to how Drill handles intervals, and the fact that the
Java 8 date/time classes are not a full replacement for Joda.)
With Joda, your format should be: "MMM dd HH:mm:ss" (Note the upper case
"H"). Try this to see if it gets you unstuck.
What we should really do is support SQL format strings. These are not standard,
but the Postgres format seem common [2]. Someone added this feature to Drill a
while back, so we must have a Postgres-to-Joda format converter in the code
somewhere we could use.
Thanks,
- Paul
[1]
https://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html
[2] https://www.postgresql.org/docs/9.1/functions-formatting.html
On Tuesday, July 16, 2019, 02:23:50 PM PDT, Charles Givre
wrote:
Hello All,
First, a big thank you Paul for updating the log regex reader to the new EVF
framework. I am having a little trouble getting it to work however...
Here is my config:
,
"ssdlog": {
"type": "logRegex",
"regex":
"(\\w{3}\\s\\d{1,2}\\s\\d{4}\\s\\d{2}:\\d{2}:\\d{2})\\s+(\\w+)\\[(\\d+)\\]:\\s(.*?(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}).*?)",
"extension": "ssdlog",
"maxErrors": 10,
"schema": [
{"fieldName":"eventDate"}
]
},
This works if I leave the schema null, however if I attempt to populate it, I
get JSON errors. This was what I originally had:
"schema" : [ {
"fieldName" : "eventDate",
"fieldType" : "TIMESTAMP",
"format" : "MMM dd hh:mm:ss"
}, {
"fieldName" : "process_name"
}, {
"fieldName" : "pid",
"fieldType" : "INT"
}, {
"fieldName" : "message"
}, {
"fieldName" : "src_ip"
} ]
which worked.
Also, I am working on updating a few format plugins and kept getting the
following error when I try to run unit tests:
at org.apache.drill.test.ClusterFixture.(ClusterFixture.java:152)
at
org.apache.drill.test.ClusterFixtureBuilder.build(ClusterFixtureBuilder.java:283)
at org.apache.drill.test.ClusterTest.startCluster(ClusterTest.java:83)
at
org.apache.drill.exec.store.excel.TestExcelFormat.setup(TestExcelFormat.java:49)
Caused by: com.typesafe.config.ConfigException$Missing: No configuration
setting found for key 'drill.exec.grace_period_ms'
at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:136)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155)
at
com.typesafe.config.impl.SimpleConfig.getConfigNumber(SimpleConfig.java:170)
at com.typesafe.config.impl.SimpleConfig.getInt(SimpleConfig.java:181)
at org.apache.drill.common.config.NestedConfig.getInt(NestedConfig.java:96)
at org.apache.drill.common.config.DrillConfig.getInt(DrillConfig.java:44)
at org.apache.drill.common.config.NestedConfig.getInt(NestedConfig.java:96)
at org.apache.drill.common.config.DrillConfig.getInt(DrillConfig.java:44)
at org.apache.drill.exec.server.Drillbit.(Drillbit.java:160)
at org.apache.drill.exec.server.Drillbit.(Drillbit.java:138)
at
org.apache.drill.test.ClusterFixture.startDrillbits(ClusterFixture.java:228)
at org.apache.drill.test.ClusterFixture.(ClusterFixture.java:146)
... 3 more
Process finished with exit code 255
I understand that I have to set the variable drill.exec.grace_period_ms, but
I'm not sure how/where to do this. Here is the beginning of my unit test code:
@ClassRule
public static final BaseDirTestWatcher dirTestWatcher = new
BaseDirTestWatcher();
@BeforeClass
public static void setup() throws Exception {
ClusterTest.startCluster(ClusterFixture.builder(dirTestWatcher).maxParallelization(1));
definePlugin();
}
private static void definePlugin() throws ExecutionSetupException {
ExcelFormatConfig sampleConfig = new ExcelFormatConfig();
// Define a temporary plugin for the "cp" storage plugin.
Drillbit drillbit = cluster.drillbit();
final StoragePluginRegistry pluginRegistry =
drillbit.getContext().getStorage();
final FileSystemPlugin plugin = (FileSystemPlugin)
pluginRegistry.getPlugin("cp");
final FileSystemConfig pluginConfig = (FileSystemConfig) plugin.getConfig();
pluginConfig.getFormats().put("sample", sampleConfig);
pluginRegistry.createOrUpdate("cp", pluginConfig, false);
}
@Test
public void testSta
EVF Log Regex Errors
Hello All,
First, a big thank you Paul for updating the log regex reader to the new EVF
framework. I am having a little trouble getting it to work however...
Here is my config:
,
"ssdlog": {
"type": "logRegex",
"regex":
"(\\w{3}\\s\\d{1,2}\\s\\d{4}\\s\\d{2}:\\d{2}:\\d{2})\\s+(\\w+)\\[(\\d+)\\]:\\s(.*?(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}).*?)",
"extension": "ssdlog",
"maxErrors": 10,
"schema": [
{"fieldName":"eventDate"}
]
},
This works if I leave the schema null, however if I attempt to populate it, I
get JSON errors. This was what I originally had:
"schema" : [ {
"fieldName" : "eventDate",
"fieldType" : "TIMESTAMP",
"format" : "MMM dd hh:mm:ss"
}, {
"fieldName" : "process_name"
}, {
"fieldName" : "pid",
"fieldType" : "INT"
}, {
"fieldName" : "message"
}, {
"fieldName" : "src_ip"
} ]
which worked.
Also, I am working on updating a few format plugins and kept getting the
following error when I try to run unit tests:
at org.apache.drill.test.ClusterFixture.(ClusterFixture.java:152)
at
org.apache.drill.test.ClusterFixtureBuilder.build(ClusterFixtureBuilder.java:283)
at org.apache.drill.test.ClusterTest.startCluster(ClusterTest.java:83)
at
org.apache.drill.exec.store.excel.TestExcelFormat.setup(TestExcelFormat.java:49)
Caused by: com.typesafe.config.ConfigException$Missing: No configuration
setting found for key 'drill.exec.grace_period_ms'
at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:136)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155)
at
com.typesafe.config.impl.SimpleConfig.getConfigNumber(SimpleConfig.java:170)
at com.typesafe.config.impl.SimpleConfig.getInt(SimpleConfig.java:181)
at
org.apache.drill.common.config.NestedConfig.getInt(NestedConfig.java:96)
at
org.apache.drill.common.config.DrillConfig.getInt(DrillConfig.java:44)
at
org.apache.drill.common.config.NestedConfig.getInt(NestedConfig.java:96)
at
org.apache.drill.common.config.DrillConfig.getInt(DrillConfig.java:44)
at org.apache.drill.exec.server.Drillbit.(Drillbit.java:160)
at org.apache.drill.exec.server.Drillbit.(Drillbit.java:138)
at
org.apache.drill.test.ClusterFixture.startDrillbits(ClusterFixture.java:228)
at org.apache.drill.test.ClusterFixture.(ClusterFixture.java:146)
... 3 more
Process finished with exit code 255
I understand that I have to set the variable drill.exec.grace_period_ms, but
I'm not sure how/where to do this. Here is the beginning of my unit test code:
@ClassRule
public static final BaseDirTestWatcher dirTestWatcher = new
BaseDirTestWatcher();
@BeforeClass
public static void setup() throws Exception {
ClusterTest.startCluster(ClusterFixture.builder(dirTestWatcher).maxParallelization(1));
definePlugin();
}
private static void definePlugin() throws ExecutionSetupException {
ExcelFormatConfig sampleConfig = new ExcelFormatConfig();
// Define a temporary plugin for the "cp" storage plugin.
Drillbit drillbit = cluster.drillbit();
final StoragePluginRegistry pluginRegistry =
drillbit.getContext().getStorage();
final FileSystemPlugin plugin = (FileSystemPlugin)
pluginRegistry.getPlugin("cp");
final FileSystemConfig pluginConfig = (FileSystemConfig) plugin.getConfig();
pluginConfig.getFormats().put("sample", sampleConfig);
pluginRegistry.createOrUpdate("cp", pluginConfig, false);
}
@Test
public void testStarQuery() throws RpcException {
String sql = "SELECT * FROM cp.`excel/test_data.xlsx` LIMIT 5";
RowSet results = client.queryBuilder().sql(sql).rowSet();
TupleMetadata expectedSchema = new SchemaBuilder()
.add("id", TypeProtos.MinorType.FLOAT8, TypeProtos.DataMode.OPTIONAL)
.add("first__name", TypeProtos.MinorType.VARCHAR,
TypeProtos.DataMode.OPTIONAL)
.add("last__name", TypeProtos.MinorType.VARCHAR,
TypeProtos.DataMode.OPTIONAL)
.add("email", TypeProtos.MinorType.VARCHAR,
TypeProtos.DataMode.OPTIONAL)
.add("gender", TypeProtos.MinorType.VARCHAR,
TypeProtos.DataMode.OPTIONAL)
.add("birthdate", TypeProtos.MinorType.VARCHAR,
TypeProtos.DataMode.OPTIONAL)
.add("balance", TypeProtos.MinorType.FLOAT8,
TypeProtos.DataMode.OPTIONAL)
.add("order__count", TypeProtos.MinorType.FLOAT8,
TypeProtos.DataMode.OPTIONAL)
.add("average__order", TypeProtos.MinorType.FLOAT8,
TypeProtos.DataMode.OPTIONAL)
.buildSchema();
RowSet expected = new Row
