cgivre commented on pull request #2359: URL: https://github.com/apache/drill/pull/2359#issuecomment-963117787
> @dzamo, the reference was to an early Saturday Night Live skit. > > Just to refocus the discussion, my question is really around configuration. When querying CSV, Parquet and the like, it is very clear where the data lies. When querying Excel, PDF, HTML, Word, "the web", it is less clear: there is some amount of data mining needed to say that it is THIS table and not THAT one, that the table spans three pages, etc. > > The question is, how does the user specify this? If it were me, I would not want to be tinkering with a JSON storage plugin config, watching my query fail with a Drill stack trace, or wondering why I got no data. Instead, I'd want a tool better suited for the task. Once I had that, if I then wanted to run at scale, I'd want to say, "Drill, just use X and consume the data." > > So, the question here is: is the JSON storage plugin config an effective way for a data scientist to mine PDF, Excel, HTML and other messy sources? I don't have an answer, I'm just asking the question. > > Again, if we had the ability to have true external plugins, then this could easily be an add-on project. Those who know PDF could go hog wild creating a good solution. But, we don't, so every specialized plugin has to be part of core Drill. Is that good? That's also a question for debate. @dzamo @paul-rogers , These are all interesting points. I'd like to focus on the config variables for a moment. The PDF reader has 2 config variables: the `mergeAllTables` and the `tableNumber`. While a user could theoretically set these globally, they really are intended to be set via the `table()` function at query time. For some reference the Excel reader has similar configs which allow a user to select different sheets within an Excel file, or define regions in a file where their data lives etc. IMHO, this flexibility is actually very good for the user, because it allows an administrator to configure global default values that make sense but also allow a user to make query-time changes so they can access their data. With respect to the fixed-width plugin (https://github.com/apache/drill/pull/2282) I actually have a different vision of how this can be used, and will post comments there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
