cgivre closed pull request #2139: DRILL-6268: Drill-on-YARN client obtains HDFS
URL incorrectly
URL: https://github.com/apache/drill/pull/2139
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
cgivre commented on pull request #2139:
URL: https://github.com/apache/drill/pull/2139#issuecomment-846099484
@paul-rogers
If I'm understanding you correctly, it sounds like the correct edits for
this PR is that I need to:
1. Modify `component.xml` as noted above.
2. Add some doc
file. Here is the default config from
`drill-on-yarn-defaults.conf`:
```
drill.yarn: {
...
dfs: {
connection: ""
app-dir: "/user/drill"
}
```
The code says:
```java
String dfsConnection
round, let's look at the [config
file](https://github.com/apache/drill/blob/master/drill-yarn/src/main/resources/org/apache/drill/yarn/core/drill-on-yarn-defaults.conf):
```
drill.yarn: {
app-name: "Drill-on-YARN"
# Settings here support a default single-node
vdiravka commented on pull request #2139:
URL: https://github.com/apache/drill/pull/2139#issuecomment-759816691
@paul-rogers Could you check please? I am going to check it on the cluster.
Are there any specific cases? Non default hadoop ports, possibly modified yarn
config file should be c
cgivre opened a new pull request #2139:
URL: https://github.com/apache/drill/pull/2139
# [DRILL-6268](https://issues.apache.org/jira/browse/DRILL-6268):
Drill-on-YARN client obtains HDFS URL Incorrectly
## Description
The Drill-on-YARN client must upload files to HDFS so that
agozhiy merged pull request #2087:
URL: https://github.com/apache/drill/pull/2087
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to th
agozhiy commented on pull request #2087:
URL: https://github.com/apache/drill/pull/2087#issuecomment-645471798
Currently it looks like this:
--
agozhiy opened a new pull request #2087:
URL: https://github.com/apache/drill/pull/2087
…strap update
Fixed navigation classes in AM templates.
# [DRILL-7749](https://issues.apache.org/jira/browse/DRILL-7749):
Drill-on-Yarn Application Master UI is broken due to bootstrap
Anton Gozhiy created DRILL-7749:
---
Summary: Drill-on-Yarn Application Master UI is broken due to
bootstrap update
Key: DRILL-7749
URL: https://issues.apache.org/jira/browse/DRILL-7749
Project: Apache
Arina Ielchiieva created DRILL-7542:
---
Summary: Fix Drill-on-Yarn logger
Key: DRILL-7542
URL: https://issues.apache.org/jira/browse/DRILL-7542
Project: Apache Drill
Issue Type: Bug
Hello Drill Devs,
My company, a major financial institution, is looking to do a POC with Drill on
our CDH cluster. Long story short, we found an issue with Drill on Yarn (DOY)
which I posted a JIRA a while ago. [1]. In going through the code, my engineer
found a TODO in the code where YARN
Charles Givre created DRILL-7149:
Summary: Kerberos Code Missing from Drill on YARN
Key: DRILL-7149
URL: https://issues.apache.org/jira/browse/DRILL-7149
Project: Apache Drill
Issue Type
Michael Dennis Uanang created DRILL-6983:
Summary: PAM Auth Enabled on Drill-On-YARN only works on YARN user
Key: DRILL-6983
URL: https://issues.apache.org/jira/browse/DRILL-6983
Project
Hi Charles,
I'm not quite sure what "dynamic queue allocation" means: all YARN containers
are allocated dynamically through YARN via queues.
It may be helpful to review how Drill-on-YARN (DoY) works. DoY does NOT attempt
to use YARN for each query. Impala tried that with Llama
Hello all,
We are trying to set up a Drill cluster on our corporate data lake. Our
cluster requires dynamic YARN queue allocation for multi-tenant environment.
Is this something that Drill supports or is there a workaround?
Thanks!
—C
Github user kkhatua commented on the issue:
https://github.com/apache/drill/pull/1239
Yes, I agree. Support folks gave me similar feedback, so I'll commit this
change in the mapr distro _IFF_ there is a request for that. YARN is already a
complex beast with numerous settings. Introduc
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1239
Just to be clear, I have no objection Drill enforcing its own cgroup limits.
My point is rather that CPU limits must be integrated with YARN, via the
DoY config file, so that the user spe
Github user kkhatua commented on the issue:
https://github.com/apache/drill/pull/1239
@paul-rogers I went through with support on this and found that the issue
is not specific to MapR. However, you make a strong argument in favor of
letting YARN handle the CGroup management rather tha
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1239
One other thing to highlight from an earlier comment. CPU is something that
the user specifies in the DoY config file. That information is passed to YARN
in container requests. This feature asks
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1239
@kkhatua, putting on my Apache hat... Apache Drill is an Apache project
that must work with other Apache projects such as Apache YARN. The Apache Drill
DoY support is designed to work well with A
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1239#discussion_r184169114
--- Diff: distribution/src/resources/yarn-drillbit.sh ---
@@ -175,4 +209,11 @@ fi
echo "`date` Starting drillbit on `hostname` under YARN, logging to
Github user kkhatua commented on the issue:
https://github.com/apache/drill/pull/1239
@paul-rogers DoY is no more a MapR-only feature, and if it helps to have
Drill self-enforce, this works. If YARN is able to enforce for Drill, the user
need not specify the settings in their `drill-e
Github user kkhatua commented on a diff in the pull request:
https://github.com/apache/drill/pull/1239#discussion_r184153040
--- Diff: distribution/src/resources/yarn-drillbit.sh ---
@@ -175,4 +209,11 @@ fi
echo "`date` Starting drillbit on `hostname` under YARN, logging to
$D
RN. Under YARN, Drill must be run in the foreground.
The original Drill-on-YARN work ensured that al this works.
---
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1239
There may be some misunderstanding of how DoY works. The only info that
users can pass to DoY is that which is in the DoY config file. We should add
arguments to that file which will be passed th
Github user kkhatua commented on the issue:
https://github.com/apache/drill/pull/1239
Thanks for that pointer, @paul-rogers ! I'll make the relevant changes and
add to this commit.
---
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1239
@kkhatua, it turns out that upstream YARN has long had effective cgroup
support per container. ( have the pleasure of sitting near the guy who
maintains that work.)There has long been a discussio
Github user kkhatua commented on a diff in the pull request:
https://github.com/apache/drill/pull/1239#discussion_r183805056
--- Diff: distribution/src/resources/yarn-drillbit.sh ---
@@ -110,6 +114,36 @@
# Enables Java GC logging. Passed from the drill.yarn.drillbit.log-gc
Github user kkhatua commented on a diff in the pull request:
https://github.com/apache/drill/pull/1239#discussion_r183732780
--- Diff: distribution/src/resources/yarn-drillbit.sh ---
@@ -110,6 +114,36 @@
# Enables Java GC logging. Passed from the drill.yarn.drillbit.log-gc
Github user Ben-Zvi commented on a diff in the pull request:
https://github.com/apache/drill/pull/1239#discussion_r183607110
--- Diff: distribution/src/resources/yarn-drillbit.sh ---
@@ -110,6 +114,36 @@
# Enables Java GC logging. Passed from the drill.yarn.drillbit.log-gc
Github user kkhatua commented on the issue:
https://github.com/apache/drill/pull/1239
@Ben-Zvi please review. QA verified that Drill-on-YARN works with this
patch.
---
GitHub user kkhatua opened a pull request:
https://github.com/apache/drill/pull/1239
CGroup Support for Drill-on-YARN
Original commit works for stand-alone Drill. During testing with
Drill-on-YARN, it was discovered that while the environment is sourced, the
Paul Rogers created DRILL-6268:
--
Summary: Drill-on-YARN client obtains HDFS URL incorrectly
Key: DRILL-6268
URL: https://issues.apache.org/jira/browse/DRILL-6268
Project: Apache Drill
Issue
Paul Rogers created DRILL-6264:
--
Summary: Incorrect log config path for Drill-on-YARN client
Key: DRILL-6264
URL: https://issues.apache.org/jira/browse/DRILL-6264
Project: Apache Drill
Issue
Github user kr-arjun commented on the issue:
https://github.com/apache/drill/pull/1011
@paul-rogers I was able to resolve this issue by workaround of setting
'yarn.timeline-service.enabled' to false ( Copied yarn-site.xml with this
property set to $DRILL_SITE directory).
Th
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1011
@arina-ielchiieva, thanks much for your help with this PR. Glad to see it
is finally in Drill master after all this time!
---
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1011
@kr-arjun, thanks for the text file. The error is related to security. DoY,
in its current for, is an "MVP": it works, but leaves off advanced features.
One of those missing features is to work w
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/1011
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1011#discussion_r172052984
--- Diff: distribution/src/assemble/bin.xml ---
@@ -323,6 +333,21 @@
src/resources/sqlline.bat
bin
+
---
Github user kr-arjun commented on the issue:
https://github.com/apache/drill/pull/1011
@paul-rogers The client error message changes look good. I did quick test
with client error message changes and could verify that error message are
logged.
> Where you using the start comm
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1011
@arina-ielchiieva, turned out that there were unneeded dependencies in the
DoY additions to the drill-root pom.xml file. Removed these and the json.org
warnings went away.
Please take a
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1011
Rebased onto latest master.
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1011#discussion_r172039445
--- Diff: distribution/src/assemble/bin.xml ---
@@ -323,6 +333,21 @@
src/resources/sqlline.bat
bin
+
--- End d
own: it is when Drill-on-YARN fails to start
the application master. There are lots of other messages for other issues such
as "Error: AM already running as Application ID: 1234" or "Failed to allocate
Drill application master."
When writing the client, I made an explici
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1011#discussion_r172023820
--- Diff: distribution/src/assemble/bin.xml ---
@@ -323,6 +333,21 @@
src/resources/sqlline.bat
bin
+
---
Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/1011
@paul-rogers when unit running tests with mapr profile, they fail because
this commit bring banned dependency:
`[INFO] --- maven-enforcer-plugin:1.3.1:enforce (avoid_bad_dependencies) @
Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/1011
@kr-arjun, I think logging full stack trace is good idea. Let's address in
new Jira.
+1, LGTM.
---
ures, only message available is 'Failed to
start Drill application master'. Do you think it would benefit troubleshooting
Drill on yarn client failures if exception stacktrace can be logged?
---
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1011
@arina-ielchiieva, do you want to give this one a committer +1? Then I'll
mark it ready-to-commit. Thanks!
---
Github user Agirish commented on the issue:
https://github.com/apache/drill/pull/1011
Looks good!
+1. Getting this into AD 1.13.0 would be great for users.
---
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1011
Rebased onto latest master.
---
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1011
Fixed the drill-common dependency as @ilooner requested.
---
Github user Agirish commented on the issue:
https://github.com/apache/drill/pull/1011
@arina-ielchiieva, sorry was held-up with something. I've just started on
this - will get back shortly.
---
? Tim has added comment
regarding the possible fix.
2. Also it would great if you can file the Jira indicating what possible
enhancements can be done. This will definitely help in future to identify main
areas of improvement for Drill on Yarn.
@Agirish
Did you have a chance to do
Github user ilooner commented on the issue:
https://github.com/apache/drill/pull/1011
@paul-rogers You need to add this dependency to your drill-yarn pom.xml
```
org.apache.drill
drill-common
${project.version}
tests
te
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1011
Failing in Travis, apparently due to test-framework issue:
```
Caused by: java.lang.ClassNotFoundException:
org.apache.drill.categories.SecurityTest
```
@ilooner, any idea wh
Github user Agirish commented on the issue:
https://github.com/apache/drill/pull/1011
@paul-rogers, I'll give it a try and update with my findings.
---
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1011
Rebased on latest master and resolved merge conflicts.
Some ZK-related classes changed. Would be good if Abhishek could do a quick
sanity test on his test cluster to make sure things stil
Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/1011
@paul-rogers based on @sachouche feedback could you please create Jira for
enhancement and also resolve conflicts in bin.xml file? Thank you in advance!
---
Github user sachouche commented on the issue:
https://github.com/apache/drill/pull/1011
+1
I have reviewed the code and overall looks good. My main feedback is that
the current implementation doesn't currently support secure clusters (at least
didn't see any logic associated with
Github user priteshm commented on the issue:
https://github.com/apache/drill/pull/1011
@sachouche @vrozov @arina-ielchiieva please review
---
GitHub user paul-rogers opened a pull request:
https://github.com/apache/drill/pull/1011
Drill 1170: Drill-on-YARN
Provides Drill integration with YARN. Runs Drill as a long-running task
under YARN. Monitors the Drill cluster, restarting failed Drillbits. Provides a
command-line
Hi All,
The Drill-on-YARN work (DRILL-1170) is drawing to a close and is ready to be
merged into master. The project resulted in about 80+ new files, 10K+ lines of
code, so this will be a bit of an effort to merge.
Drill-on-YARN is actually independent of the rest of Drill. Except for changes
for CPU.
If we assume the current threading model, as you suggest, we’ll still be fine
in terms of CPU usage. The current model can be a bit exuberant in its use of
CPU, but, cgroups will ensure that Drill cannot exceed the YARN-imposed CPU
limit.
Once the Drill-on-YARN work gets a bit further
; > memory (or release excess memory). Presumably the app or a user must
> decide
> > to request more memory. For example, the admin might dial up Drill memory
> > during the day when the marketing folks are running queries, but dial it
> > back at night when mostly batch jobs run.
&
search on YARN did not reveal anything in this
> vein, so we’ll have to research this idea a bit more. I wonder, though, if
> Drill could actually handle fast-moving allocation changes; change on the
> order of the lifetime of a query seems more achievable (that is, on the
> order of m
e (that is, on the order of minutes to
hours).
In short, it seems we have quite a few tasks ahead in the area of memory
management. Each seems achievable, but each requires work. The Drill-on-YARN
project is just a start: it helps the admin allocate memory between Drill and
other apps.
Thank
ees would rank higher.
>
> The Llama approach seems to be to much of a "house of cards" to me to be
> viable, and I worry that long term it may not be best for a product like
> Drill. Our goal I think should be to play nice with others, if our core
> philosophy in integ
r a product like
Drill. Our goal I think should be to play nice with others, if our core
philosophy in integration is playing nice with others, it will only help
adoption and people giving it a try. So back to Drill on Yarn (natively)...
A few questions around this. You mention that resou
her one of the existing
>> YARN
>>> frameworks such as Slider would be useful? Does anyone on the list have
>> any
>>> feedback on the relative merits of these technologies?
>>>
>>> Again, glad to see someone picking this up.
>>>
>&
> --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Tue, Mar 22, 2016 at 4:58 PM, Paul Rogers
> wrote:
> >
> >> Hi All,
> >>
> >> I’m a new member of the Drill Team here at MapR. We’d like to take a
> look
> >>
Happy to help. I will stay involved on the Yarn side too, my hope is any
improvements to drill to facilitate a benefit for drill on yarn can be
abstracted and not just be a drill on yarn feature, but instead, create
hooks to do things (like draining nodes we wish to shutdown, or scale
memory and
separate.
We’re looking at YARN first primarily because it seems to come up more often:
everyone who uses MapReduce is familiar with YARN. Mesos seems a more advanced
solution adopted by more experienced folks such as yourself. We’re hoping that
the lessons we learn about managing Drill on YARN can
feedback on the relative merits of these technologies?
> >
> > Again, glad to see someone picking this up.
> >
> > Jacques
> >
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Tue, Mar 22, 2016 at 4:58 PM, Paul
se technologies?
>
> Again, glad to see someone picking this up.
>
> Jacques
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Tue, Mar 22, 2016 at 4:58 PM, Paul Rogers wrote:
>
>> Hi All,
>>
>> I’m a new member of the Dr
22, 2016 at 4:58 PM, Paul Rogers wrote:
> Hi All,
>
> I’m a new member of the Drill Team here at MapR. We’d like to take a look
> at running Drill on YARN for production customers. JIRA suggests some early
> work may have been done (DRILL-142 <
> https://issues.apache.org/
Hi All,
I’m a new member of the Drill Team here at MapR. We’d like to take a look at
running Drill on YARN for production customers. JIRA suggests some early work
may have been done (DRILL-142
<https://issues.apache.org/jira/browse/DRILL-142>, DRILL-1170
<https://issues.apache.org/ji
78 matches
Mail list logo
