Re: Query XML-files with Drill

[Magnus Pierre]

Hello Per,

I (not in any way related to MapR engineering) used to develop an idea of a 
plugin for XML that is using on-the-fly behind the curtains conversion to JSON 
to utilize all the goodies Drill has for JSON but due to a stumbling block (XML 
needs the support of a very dynamic and flexible JSON schema due to the nature 
of XML that Drill only can support if UNION type is activated) and that is for 
good reasons filtered out for embeddedContent which my plugin provides to the 
JsonRecordReader:

From: JdbcRecordReader.java:
this.unionEnabled = embeddedContent == null && 
fragmentContext.getOptions().getOption(ExecConstants.ENABLE_UNION_TYPE);

I decided to pause the efforts. FYI: The Drill union type had issues with 
embeddedContent that were somewhat random and very hard to debug which I never 
saw when handing the generated JSON as a file. I am sure that someone more 
competent in the Drill code would be able to figure out what is happening. :)

Your best alternative as-is today is probably to do processing of XML in Spark 
the XML plugin for Spark is quite competent, or do a simple XML to json 
conversion in Spark and materialize as JSON documents that you then query with 
Drill. 

Regards,
Magnus



> 26 aug 2016 kl. 08:36 skrev Per Weinberger :
> 
> Hi,
> 
> I'm looking for examples or information on how to query xml-files with Drill. 
> I working with somewhat large (100+ mb) xml-files and would like to query 
> them in-situ. Are there any examples or information regarding this? I would 
> think that this would be a fairly common thing to do, but there is very 
> little regarding this on Google or Stack overflow.
> 
> Cheers,
> Per Weinberger
> ***
> CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the exclusive 
> and confidential use of the intended recipient and may constitute non-public 
> information. If you received this e-mail in error, disclosing, copying, 
> distributing or taking any action in reliance of this e-mail is strictly 
> prohibited and may be unlawful. Instead, please notify us immediately by 
> return e-mail and promptly delete this message and its attachments from your 
> computer system. We do not waive any work product or other applicable legal 
> privilege(s) by the transmission of this message.
> ***

Query XML-files with Drill

[Per Weinberger]

Hi,

I'm looking for examples or information on how to query xml-files with Drill. I 
working with somewhat large (100+ mb) xml-files and would like to query them 
in-situ. Are there any examples or information regarding this? I would think 
that this would be a fairly common thing to do, but there is very little 
regarding this on Google or Stack overflow.

Cheers,
Per Weinberger
***
CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the exclusive 
and confidential use of the intended recipient and may constitute non-public 
information. If you received this e-mail in error, disclosing, copying, 
distributing or taking any action in reliance of this e-mail is strictly 
prohibited and may be unlawful. Instead, please notify us immediately by return 
e-mail and promptly delete this message and its attachments from your computer 
system. We do not waive any work product or other applicable legal privilege(s) 
by the transmission of this message.
***

Re: move drill log directory to HDFS

[Paul Rogers]

I don’t believe it is possible to store Drill logs in HDFS using an HDFS URL. 
The log files are set up via environment variables in the drill-config.sh 
script and are expected to be normal file system files.

However, if you have the ability to NFS mount your distributed file system, 
then you can easily accomplish your goal. (MapR MFS is such an example, others 
may support this as well.) Let’s say that you mount your DFS at /mnt/dfs. Then, 
you can add the following to your drill-env.sh script:

export DRILL_LOG_DIR=/mnt/dfs/drill/`hostname`

The bit about hostname is needed to put the logs for each Drillbit into a 
separate directory. Feel free to use any other technique you like to label the 
directory.

As an alternative, we are making good progress in finalizing a YARN hosting 
solution for Drill. Once this is available (probably Drill 1.9), then YARN will 
take care of collecting your Drill log files.

- Paul

> On Aug 26, 2016, at 6:46 AM, Andries Engelbrecht  
> wrote:
> 
> https://drill.apache.org/docs/persistent-configuration-storage/ 
> 
> 
> 
> --Andries
> 
>> On Aug 26, 2016, at 4:25 AM, Anup Tiwari  wrote:
>> 
>> Also please note that I have tried below in all node's drill-env.sh but its
>> not working.
>> 
>> export DRILL_LOG_DIR="hdfs://namenode:9000/tmp/drilllogs/"
>> 
>> 
>> Regards,
>> *Anup Tiwari*
>> 
>> 
>> On Fri, Aug 26, 2016 at 4:06 PM, Anup Tiwari 
>> wrote:
>> 
>>> Hi All,
>>> 
>>> We are trying to move drill logs directory from local file system to HDFS
>>> so that we can refer only one location rather than each node's log
>>> directory.
>>> 
>>> Can anyone help me on this?
>>> 
>>> Regards,
>>> *Anup Tiwari*
>>> 
>>> 
> 

Re: Facing issue with drill web UI.

[Jinfeng Ni]

That's probably because your login is not an admin user, or does not
belong to a group having the admin authorization. Take a look at [1]

[1] https://drill.apache.org/docs/configuring-web-console-and-rest-api-security/

On Fri, Aug 26, 2016 at 12:10 AM, Sonali Ghorpade
 wrote:
> Hi,
>
> I have configured custom authentication in dill and modified the
> drill-override.conf file according to the documentation mentioned in below
> link.
>
> https://drill.apache.org/docs/configuring-user-authentication/
> my authentication jar is working fine. but after modifying the
> drill-override.conf the Storage, Threads, Logs tab in drill web UI is not
> seen. can anyone please help me with this issue. my content of
> rill-override.conf are
> drill.exec {
>
> cluster-id: "XXX",
>
> zk.connect: "",
> security.user.auth {
>
> enabled: true,
>
> packages += "org.ebi.drill.security",
>
> impl: "CustomAuthenticatorType"
>
> }
>
> }
> if i comment the security.user.auth the Storage, Threads, Logs tab in drill
> web UI is shown.please refer the attachment

Re: move drill log directory to HDFS

[Andries Engelbrecht]

https://drill.apache.org/docs/persistent-configuration-storage/ 



--Andries

> On Aug 26, 2016, at 4:25 AM, Anup Tiwari  wrote:
> 
> Also please note that I have tried below in all node's drill-env.sh but its
> not working.
> 
> export DRILL_LOG_DIR="hdfs://namenode:9000/tmp/drilllogs/"
> 
> 
> Regards,
> *Anup Tiwari*
> 
> 
> On Fri, Aug 26, 2016 at 4:06 PM, Anup Tiwari 
> wrote:
> 
>> Hi All,
>> 
>> We are trying to move drill logs directory from local file system to HDFS
>> so that we can refer only one location rather than each node's log
>> directory.
>> 
>> Can anyone help me on this?
>> 
>> Regards,
>> *Anup Tiwari*
>> 
>> 

Facing issue with drill web UI.

[Sonali Ghorpade]

Hi,

I have configured custom authentication in dill and modified the
drill-override.conf file according to the documentation mentioned in below
link.

https://drill.apache.org/docs/configuring-user-authentication/​
my authentication jar is working fine. but after modifying the
drill-override.conf the Storage, Threads, Logs tab in drill web UI is not
seen. can anyone please help me with this issue. my content of
rill-override.conf are
drill.exec {

cluster-id: "XXX",

zk.connect: "",
security.user.auth {

enabled: true,

packages += "org.ebi.drill.security",

impl: "CustomAuthenticatorType"

}

}
if i comment the security.user.auth the Storage, Threads, Logs tab in drill
web UI is shown.please refer the attachment

Re: move drill log directory to HDFS

[Anup Tiwari]

Also please note that I have tried below in all node's drill-env.sh but its
not working.

export DRILL_LOG_DIR="hdfs://namenode:9000/tmp/drilllogs/"


Regards,
*Anup Tiwari*


On Fri, Aug 26, 2016 at 4:06 PM, Anup Tiwari 
wrote:

> Hi All,
>
> We are trying to move drill logs directory from local file system to HDFS
> so that we can refer only one location rather than each node's log
> directory.
>
> Can anyone help me on this?
>
> Regards,
> *Anup Tiwari*
>
>

move drill log directory to HDFS

[Anup Tiwari]

Hi All,

We are trying to move drill logs directory from local file system to HDFS
so that we can refer only one location rather than each node's log
directory.

Can anyone help me on this?

Regards,
*Anup Tiwari*