[jira] [Created] (DRILL-7030) Make format plugins fully plugable
Arina Ielchiieva created DRILL-7030:
---
Summary: Make format plugins fully plugable
Key: DRILL-7030
URL: https://issues.apache.org/jira/browse/DRILL-7030
Project: Apache Drill
Issue Type: Improvement
Affects Versions: 1.15.0
Environment: Discussion on the mailing list -
[https://lists.apache.org/thread.html/0c7de9c23ee9a8e18f8548ae0a323284cf1311b9570bd77ba544f63d@%3Cdev.drill.apache.org%3E]
{noformat}
Before we were adding new formats / plugins into the exec module. Eventually we
came up to the point that exec package size is growing and adding plugin and
format contributions is better to separate out in the different module.
Now we have contrib module where we add such contributions. Plugins are
pluggable, there are added automatically by means of having drill-module.conf
file which points to the scanning packages.
Format plugins are using the same approach, the only problem is that they are
not added into bootstrap-storage-plugins.json. So when adding new format
plugin, in order for it to automatically appear in Drill Web UI, developer has
to update bootstrap file which is in the exec module.
My suggestion we implement some functionality that would merge format config
with the bootstrap one. For example, each plugin would have to have
bootstrap-format.json file with the information to which plugin format should
be added (structure the same as in bootstrap-storage-plugins.json):
Example:
{
"storage":{
dfs: {
formats: {
"psv" : {
type: "msgpack",
extensions: [ "mp" ]
}
}
}
}
}
Then during Drill start up such bootstrap-format.json files will be merged with
bootstrap-storage-plugins.json.
{noformat}
Reporter: Arina Ielchiieva
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-7031) Add Travis job that runs protobuf generation command and checks if all protobufs are up-to-date
Arina Ielchiieva created DRILL-7031: --- Summary: Add Travis job that runs protobuf generation command and checks if all protobufs are up-to-date Key: DRILL-7031 URL: https://issues.apache.org/jira/browse/DRILL-7031 Project: Apache Drill Issue Type: Improvement Affects Versions: 1.15.0 Reporter: Arina Ielchiieva Add Travis job that runs protobuf generation command and checks if all protobufs are up-to-date. Should cover Java and C++ protobufs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-7030) Make format plugins fully plugable
[
https://issues.apache.org/jira/browse/DRILL-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-7030:
Description:
Discussion on the mailing list -
[https://lists.apache.org/thread.html/0c7de9c23ee9a8e18f8548ae0a323284cf1311b9570bd77ba544f63d@%3Cdev.drill.apache.org%3E]
{noformat}
Before we were adding new formats / plugins into the exec module. Eventually we
came up to the point that exec package size is growing and adding plugin and
format contributions is better to separate out in the different module.
Now we have contrib module where we add such contributions. Plugins are
pluggable, there are added automatically by means of having drill-module.conf
file which points to the scanning packages.
Format plugins are using the same approach, the only problem is that they are
not added into bootstrap-storage-plugins.json. So when adding new format
plugin, in order for it to automatically appear in Drill Web UI, developer has
to update bootstrap file which is in the exec module.
My suggestion we implement some functionality that would merge format config
with the bootstrap one. For example, each plugin would have to have
bootstrap-format.json file with the information to which plugin format should
be added (structure the same as in bootstrap-storage-plugins.json):
Example:
{
"storage":{
dfs: {
formats: {
"psv" : {
type: "msgpack",
extensions: [ "mp" ]
}
}
}
}
}
Then during Drill start up such bootstrap-format.json files will be merged with
bootstrap-storage-plugins.json.{noformat}
> Make format plugins fully plugable
> --
>
> Key: DRILL-7030
> URL: https://issues.apache.org/jira/browse/DRILL-7030
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.15.0
> Environment: Discussion on the mailing list -
> [https://lists.apache.org/thread.html/0c7de9c23ee9a8e18f8548ae0a323284cf1311b9570bd77ba544f63d@%3Cdev.drill.apache.org%3E]
> {noformat}
> Before we were adding new formats / plugins into the exec module. Eventually
> we came up to the point that exec package size is growing and adding plugin
> and format contributions is better to separate out in the different module.
> Now we have contrib module where we add such contributions. Plugins are
> pluggable, there are added automatically by means of having drill-module.conf
> file which points to the scanning packages.
> Format plugins are using the same approach, the only problem is that they are
> not added into bootstrap-storage-plugins.json. So when adding new format
> plugin, in order for it to automatically appear in Drill Web UI, developer
> has to update bootstrap file which is in the exec module.
> My suggestion we implement some functionality that would merge format config
> with the bootstrap one. For example, each plugin would have to have
> bootstrap-format.json file with the information to which plugin format should
> be added (structure the same as in bootstrap-storage-plugins.json):
> Example:
> {
> "storage":{
> dfs: {
> formats: {
> "psv" : {
> type: "msgpack",
> extensions: [ "mp" ]
> }
> }
> }
> }
> }
> Then during Drill start up such bootstrap-format.json files will be merged
> with bootstrap-storage-plugins.json.
> {noformat}
>
>
>Reporter: Arina Ielchiieva
>Priority: Major
>
> Discussion on the mailing list -
> [https://lists.apache.org/thread.html/0c7de9c23ee9a8e18f8548ae0a323284cf1311b9570bd77ba544f63d@%3Cdev.drill.apache.org%3E]
> {noformat}
> Before we were adding new formats / plugins into the exec module. Eventually
> we came up to the point that exec package size is growing and adding plugin
> and format contributions is better to separate out in the different module.
> Now we have contrib module where we add such contributions. Plugins are
> pluggable, there are added automatically by means of having drill-module.conf
> file which points to the scanning packages.
> Format plugins are using the same approach, the only problem is that they are
> not added into bootstrap-storage-plugins.json. So when adding new format
> plugin, in order for it to automatically appear in Drill Web UI, developer
> has to update bootstrap file which is in the exec module.
> My suggestion we implement some functionality that would merge format config
> with the bootstrap one. For example, each plugin would have to have
> bootstrap-format.json file with the information to which plugin format should
> be added (structure the same as in bootstrap-storage-plugins.json):
> Example:
> {
> "storage":{
> dfs: {
> formats: {
> "psv" : {
> type: "msgpack",
> extensions: [ "mp" ]
> }
> }
> }
> }
> }
> Then duri
[jira] [Updated] (DRILL-7030) Make format plugins fully plugable
[
https://issues.apache.org/jira/browse/DRILL-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-7030:
Environment: (was: Discussion on the mailing list -
[https://lists.apache.org/thread.html/0c7de9c23ee9a8e18f8548ae0a323284cf1311b9570bd77ba544f63d@%3Cdev.drill.apache.org%3E]
{noformat}
Before we were adding new formats / plugins into the exec module. Eventually we
came up to the point that exec package size is growing and adding plugin and
format contributions is better to separate out in the different module.
Now we have contrib module where we add such contributions. Plugins are
pluggable, there are added automatically by means of having drill-module.conf
file which points to the scanning packages.
Format plugins are using the same approach, the only problem is that they are
not added into bootstrap-storage-plugins.json. So when adding new format
plugin, in order for it to automatically appear in Drill Web UI, developer has
to update bootstrap file which is in the exec module.
My suggestion we implement some functionality that would merge format config
with the bootstrap one. For example, each plugin would have to have
bootstrap-format.json file with the information to which plugin format should
be added (structure the same as in bootstrap-storage-plugins.json):
Example:
{
"storage":{
dfs: {
formats: {
"psv" : {
type: "msgpack",
extensions: [ "mp" ]
}
}
}
}
}
Then during Drill start up such bootstrap-format.json files will be merged with
bootstrap-storage-plugins.json.
{noformat}
)
> Make format plugins fully plugable
> --
>
> Key: DRILL-7030
> URL: https://issues.apache.org/jira/browse/DRILL-7030
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.15.0
>Reporter: Arina Ielchiieva
>Priority: Major
>
> Discussion on the mailing list -
> [https://lists.apache.org/thread.html/0c7de9c23ee9a8e18f8548ae0a323284cf1311b9570bd77ba544f63d@%3Cdev.drill.apache.org%3E]
> {noformat}
> Before we were adding new formats / plugins into the exec module. Eventually
> we came up to the point that exec package size is growing and adding plugin
> and format contributions is better to separate out in the different module.
> Now we have contrib module where we add such contributions. Plugins are
> pluggable, there are added automatically by means of having drill-module.conf
> file which points to the scanning packages.
> Format plugins are using the same approach, the only problem is that they are
> not added into bootstrap-storage-plugins.json. So when adding new format
> plugin, in order for it to automatically appear in Drill Web UI, developer
> has to update bootstrap file which is in the exec module.
> My suggestion we implement some functionality that would merge format config
> with the bootstrap one. For example, each plugin would have to have
> bootstrap-format.json file with the information to which plugin format should
> be added (structure the same as in bootstrap-storage-plugins.json):
> Example:
> {
> "storage":{
> dfs: {
> formats: {
> "psv" : {
> type: "msgpack",
> extensions: [ "mp" ]
> }
> }
> }
> }
> }
> Then during Drill start up such bootstrap-format.json files will be merged
> with bootstrap-storage-plugins.json.{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-7032) Ignore corrupt rows in a PCAP file
Giovanni Conte created DRILL-7032: - Summary: Ignore corrupt rows in a PCAP file Key: DRILL-7032 URL: https://issues.apache.org/jira/browse/DRILL-7032 Project: Apache Drill Issue Type: Improvement Components: Functions - Drill Affects Versions: 1.15.0 Environment: OS: Ubuntu 18.4 Drill version: 1.15.0 Java(TM) SE Runtime Environment (build 1.8.0_191-b12) Reporter: Giovanni Conte Attachments: testv1.pcap Would be useful for Drill to have some ability to ignore corrupt rows in a PCAP file instead of trow the java exception. This is because there are many pcap files with corrupted lines and this funcionality will avoid to do a pre-fixing of the packet-captures (example attached file). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-7030) Make format plugins fully plugable
[
https://issues.apache.org/jira/browse/DRILL-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16762010#comment-16762010
]
Paul Rogers commented on DRILL-7030:
Please also ensure that the unit testing tools available in exec are also
available in contrib. In particular, a user should be able to write a unit
test, run it from their IDE, and step through their plugin code. If change is
needed, the needed build should be within the IDE, no need to invoke maven.
That is, contrib should enable the rapid edit-compile-debug cycle that we've
created in exec.
> Make format plugins fully plugable
> --
>
> Key: DRILL-7030
> URL: https://issues.apache.org/jira/browse/DRILL-7030
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.15.0
>Reporter: Arina Ielchiieva
>Priority: Major
>
> Discussion on the mailing list -
> [https://lists.apache.org/thread.html/0c7de9c23ee9a8e18f8548ae0a323284cf1311b9570bd77ba544f63d@%3Cdev.drill.apache.org%3E]
> {noformat}
> Before we were adding new formats / plugins into the exec module. Eventually
> we came up to the point that exec package size is growing and adding plugin
> and format contributions is better to separate out in the different module.
> Now we have contrib module where we add such contributions. Plugins are
> pluggable, there are added automatically by means of having drill-module.conf
> file which points to the scanning packages.
> Format plugins are using the same approach, the only problem is that they are
> not added into bootstrap-storage-plugins.json. So when adding new format
> plugin, in order for it to automatically appear in Drill Web UI, developer
> has to update bootstrap file which is in the exec module.
> My suggestion we implement some functionality that would merge format config
> with the bootstrap one. For example, each plugin would have to have
> bootstrap-format.json file with the information to which plugin format should
> be added (structure the same as in bootstrap-storage-plugins.json):
> Example:
> {
> "storage":{
> dfs: {
> formats: {
> "psv" : {
> type: "msgpack",
> extensions: [ "mp" ]
> }
> }
> }
> }
> }
> Then during Drill start up such bootstrap-format.json files will be merged
> with bootstrap-storage-plugins.json.{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6960) Auto Limit Wrapping should not apply to non-select query
[
https://issues.apache.org/jira/browse/DRILL-6960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua updated DRILL-6960:
Labels: user-experience (was: doc-impacting user-experience)
> Auto Limit Wrapping should not apply to non-select query
>
>
> Key: DRILL-6960
> URL: https://issues.apache.org/jira/browse/DRILL-6960
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.16.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Labels: user-experience
> Fix For: 1.16.0
>
>
> [~IhorHuzenko] pointed out that DRILL-6050 can cause submission of queries
> with incorrect syntax.
> For example, when user enters {{SHOW DATABASES}}' and after limitation
> wrapping {{SELECT * FROM (SHOW DATABASES) LIMIT 10}} will be posted.
> This results into parsing errors, like:
> {{Query Failed: An Error Occurred
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "( show" at line 2, column 15. Was expecting one of:
> ... }}.
> The fix should involve a javascript check for all non-select queries and not
> apply the LIMIT wrap for those queries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-7032) Ignore corrupt rows in a PCAP file
[ https://issues.apache.org/jira/browse/DRILL-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Givre reassigned DRILL-7032: Assignee: Charles Givre > Ignore corrupt rows in a PCAP file > -- > > Key: DRILL-7032 > URL: https://issues.apache.org/jira/browse/DRILL-7032 > Project: Apache Drill > Issue Type: Improvement > Components: Functions - Drill >Affects Versions: 1.15.0 > Environment: OS: Ubuntu 18.4 > Drill version: 1.15.0 > Java(TM) SE Runtime Environment (build 1.8.0_191-b12) >Reporter: Giovanni Conte >Assignee: Charles Givre >Priority: Major > Attachments: testv1.pcap > > > Would be useful for Drill to have some ability to ignore corrupt rows in a > PCAP file instead of trow the java exception. > This is because there are many pcap files with corrupted lines and this > funcionality will avoid to do a pre-fixing of the packet-captures (example > attached file). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
