AWS EMR bootstrap script to install and configure Drill
I was wondering if anyone had an up to date bootstrap script that installs and configures Drill on an AWS EMR cluster. I'm looking for one that will work with the stock Amazon EMR version and not a MapR EMR cluster. I've had too much trouble in the past with the MapR EMR bootstrap scripts themselves failing. Nowadays I stick to the stock Amazon EMR. I've done some web searching, but all I can seem to find are pretty old, outdated or in some cases non-existent scripts. So I know people have tried it in the past. I'm very new to Drill, so having something to start with at least would be a huge help. Thanks, Dave
Re: AWS EMR bootstrap script to install and configure Drill
Dave, This is a little bit old/outdated but may be better than others you found: https://github.com/commercehub-oss/watershed/blob/master/watershed/resources/s3/emr/exec/setup_drill We haven’t gotten around to updating the overall Watershed project for EMR 4, which is managed differently, nor more recent versions of Drill. Mostly because it still meets our needs. But that has discouraged us from announcing the project, so I’d really like to get it done one of these days. Paul Mogren On 6/20/16, 10:29 AM, "David Kincaid" wrote: I was wondering if anyone had an up to date bootstrap script that installs and configures Drill on an AWS EMR cluster. I'm looking for one that will work with the stock Amazon EMR version and not a MapR EMR cluster. I've had too much trouble in the past with the MapR EMR bootstrap scripts themselves failing. Nowadays I stick to the stock Amazon EMR. I've done some web searching, but all I can seem to find are pretty old, outdated or in some cases non-existent scripts. So I know people have tried it in the past. I'm very new to Drill, so having something to start with at least would be a huge help. Thanks, Dave
Re: AWS EMR bootstrap script to install and configure Drill
Thanks, Paul. This does look like a good place to start. Unfortunately, it fails right off the bat due to the emr/common library not being available. Not being a Ruby guy, I'm not sure where to go from here. Is there some package that I can easily install to get that library? - Dave Here's the error for reference: /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- emr/common (LoadError) from /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in `require' from ./setup-drill:39:in `' On Wed, Jun 29, 2016 at 4:04 PM, Paul Mogren wrote: > Dave, > > This is a little bit old/outdated but may be better than others you found: > > https://github.com/commercehub-oss/watershed/blob/master/watershed/resources/s3/emr/exec/setup_drill > > We haven’t gotten around to updating the overall Watershed project for EMR > 4, which is managed differently, nor more recent versions of Drill. Mostly > because it still meets our needs. But that has discouraged us from > announcing the project, so I’d really like to get it done one of these days. > > Paul Mogren > > > On 6/20/16, 10:29 AM, "David Kincaid" wrote: > > I was wondering if anyone had an up to date bootstrap script that installs > and configures Drill on an AWS EMR cluster. I'm looking for one that will > work with the stock Amazon EMR version and not a MapR EMR cluster. I've had > too much trouble in the past with the MapR EMR bootstrap scripts themselves > failing. Nowadays I stick to the stock Amazon EMR. I've done some web > searching, but all I can seem to find are pretty old, outdated or in some > cases non-existent scripts. So I know people have tried it in the past. I'm > very new to Drill, so having something to start with at least would be a > huge help. > > Thanks, > > Dave > > >
Re: AWS EMR bootstrap script to install and configure Drill
Did you by chance attempt to run it outside of EMR? AWS is known to provide a local install of EMR integration libraries on the nodes, but not publish them to OSS repositories. This was derived from the old https://github.com/awslabs/emr-bootstrap-actions/tree/master/drill and worked for us fairly recently. > On Jul 1, 2016, at 12:54 PM, David Kincaid wrote: > > Thanks, Paul. This does look like a good place to start. Unfortunately, it > fails right off the bat due to the emr/common library not being available. > Not being a Ruby guy, I'm not sure where to go from here. Is there some > package that I can easily install to get that library? > > - Dave > > Here's the error for reference: > /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in > `require': cannot load such file -- emr/common (LoadError) > from > /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in > `require' > from ./setup-drill:39:in `' > > > On Wed, Jun 29, 2016 at 4:04 PM, Paul Mogren > wrote: > >> Dave, >> >> This is a little bit old/outdated but may be better than others you found: >> >> https://github.com/commercehub-oss/watershed/blob/master/watershed/resources/s3/emr/exec/setup_drill >> >> We haven’t gotten around to updating the overall Watershed project for EMR >> 4, which is managed differently, nor more recent versions of Drill. Mostly >> because it still meets our needs. But that has discouraged us from >> announcing the project, so I’d really like to get it done one of these days. >> >> Paul Mogren >> >> >> On 6/20/16, 10:29 AM, "David Kincaid" wrote: >> >> I was wondering if anyone had an up to date bootstrap script that installs >> and configures Drill on an AWS EMR cluster. I'm looking for one that will >> work with the stock Amazon EMR version and not a MapR EMR cluster. I've had >> too much trouble in the past with the MapR EMR bootstrap scripts themselves >> failing. Nowadays I stick to the stock Amazon EMR. I've done some web >> searching, but all I can seem to find are pretty old, outdated or in some >> cases non-existent scripts. So I know people have tried it in the past. I'm >> very new to Drill, so having something to start with at least would be a >> huge help. >> >> Thanks, >> >> Dave >> >> >>
Re: AWS EMR bootstrap script to install and configure Drill
I tried to run it from the shell (ssh'd in) on the master node of a newly spun up EMR cluster which used the AWS EMR 4.x. On Fri, Jul 1, 2016 at 10:33 PM, Paul Mogren wrote: > Did you by chance attempt to run it outside of EMR? AWS is known to > provide a local install of EMR integration libraries on the nodes, but not > publish them to OSS repositories. > > This was derived from the old > https://github.com/awslabs/emr-bootstrap-actions/tree/master/drill and > worked for us fairly recently. > > > > > On Jul 1, 2016, at 12:54 PM, David Kincaid > wrote: > > > > Thanks, Paul. This does look like a good place to start. Unfortunately, > it > > fails right off the bat due to the emr/common library not being > available. > > Not being a Ruby guy, I'm not sure where to go from here. Is there some > > package that I can easily install to get that library? > > > > - Dave > > > > Here's the error for reference: > > /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in > > `require': cannot load such file -- emr/common (LoadError) > > from > > /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in > > `require' > > from ./setup-drill:39:in `' > > > > > > On Wed, Jun 29, 2016 at 4:04 PM, Paul Mogren > > wrote: > > > >> Dave, > >> > >> This is a little bit old/outdated but may be better than others you > found: > >> > >> > https://github.com/commercehub-oss/watershed/blob/master/watershed/resources/s3/emr/exec/setup_drill > >> > >> We haven’t gotten around to updating the overall Watershed project for > EMR > >> 4, which is managed differently, nor more recent versions of Drill. > Mostly > >> because it still meets our needs. But that has discouraged us from > >> announcing the project, so I’d really like to get it done one of these > days. > >> > >> Paul Mogren > >> > >> > >> On 6/20/16, 10:29 AM, "David Kincaid" wrote: > >> > >> I was wondering if anyone had an up to date bootstrap script that > installs > >> and configures Drill on an AWS EMR cluster. I'm looking for one that > will > >> work with the stock Amazon EMR version and not a MapR EMR cluster. I've > had > >> too much trouble in the past with the MapR EMR bootstrap scripts > themselves > >> failing. Nowadays I stick to the stock Amazon EMR. I've done some web > >> searching, but all I can seem to find are pretty old, outdated or in > some > >> cases non-existent scripts. So I know people have tried it in the past. > I'm > >> very new to Drill, so having something to start with at least would be a > >> huge help. > >> > >> Thanks, > >> > >> Dave > >> > >> > >> >
Re: AWS EMR bootstrap script to install and configure Drill
Could be a EMR 3 vs 4 difference. > On Jul 2, 2016, at 8:51 AM, David Kincaid wrote: > > I tried to run it from the shell (ssh'd in) on the master node of a newly > spun up EMR cluster which used the AWS EMR 4.x. > > On Fri, Jul 1, 2016 at 10:33 PM, Paul Mogren > wrote: > >> Did you by chance attempt to run it outside of EMR? AWS is known to >> provide a local install of EMR integration libraries on the nodes, but not >> publish them to OSS repositories. >> >> This was derived from the old >> https://github.com/awslabs/emr-bootstrap-actions/tree/master/drill and >> worked for us fairly recently. >> >> >> On Jul 1, 2016, at 12:54 PM, David Kincaid >>> wrote: >>> >>> Thanks, Paul. This does look like a good place to start. Unfortunately, >> it >>> fails right off the bat due to the emr/common library not being >> available. >>> Not being a Ruby guy, I'm not sure where to go from here. Is there some >>> package that I can easily install to get that library? >>> >>> - Dave >>> >>> Here's the error for reference: >>> /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in >>> `require': cannot load such file -- emr/common (LoadError) >>> from >>> /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in >>> `require' >>> from ./setup-drill:39:in `' >>> >>> >>> On Wed, Jun 29, 2016 at 4:04 PM, Paul Mogren >>> wrote: >>> Dave, This is a little bit old/outdated but may be better than others you >> found: >> https://github.com/commercehub-oss/watershed/blob/master/watershed/resources/s3/emr/exec/setup_drill We haven’t gotten around to updating the overall Watershed project for >> EMR 4, which is managed differently, nor more recent versions of Drill. >> Mostly because it still meets our needs. But that has discouraged us from announcing the project, so I’d really like to get it done one of these >> days. Paul Mogren On 6/20/16, 10:29 AM, "David Kincaid" wrote: I was wondering if anyone had an up to date bootstrap script that >> installs and configures Drill on an AWS EMR cluster. I'm looking for one that >> will work with the stock Amazon EMR version and not a MapR EMR cluster. I've >> had too much trouble in the past with the MapR EMR bootstrap scripts >> themselves failing. Nowadays I stick to the stock Amazon EMR. I've done some web searching, but all I can seem to find are pretty old, outdated or in >> some cases non-existent scripts. So I know people have tried it in the past. >> I'm very new to Drill, so having something to start with at least would be a huge help. Thanks, Dave >>
Re: AWS EMR bootstrap script to install and configure Drill
Yes, very well could be. A lot of big changes with version 4. I'll see if I can figure out where that library may have gone. They have added Zookeeper as a supported application that EMR can install at startup, so at least we should be able to eliminate the Zookeeper install from the bootstrap script. On Sat, Jul 2, 2016 at 12:00 PM, Paul Mogren wrote: > Could be a EMR 3 vs 4 difference. > > > > On Jul 2, 2016, at 8:51 AM, David Kincaid > wrote: > > > > I tried to run it from the shell (ssh'd in) on the master node of a newly > > spun up EMR cluster which used the AWS EMR 4.x. > > > > On Fri, Jul 1, 2016 at 10:33 PM, Paul Mogren > > wrote: > > > >> Did you by chance attempt to run it outside of EMR? AWS is known to > >> provide a local install of EMR integration libraries on the nodes, but > not > >> publish them to OSS repositories. > >> > >> This was derived from the old > >> https://github.com/awslabs/emr-bootstrap-actions/tree/master/drill and > >> worked for us fairly recently. > >> > >> > >> > On Jul 1, 2016, at 12:54 PM, David Kincaid > >>> wrote: > >>> > >>> Thanks, Paul. This does look like a good place to start. Unfortunately, > >> it > >>> fails right off the bat due to the emr/common library not being > >> available. > >>> Not being a Ruby guy, I'm not sure where to go from here. Is there some > >>> package that I can easily install to get that library? > >>> > >>> - Dave > >>> > >>> Here's the error for reference: > >>> > /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in > >>> `require': cannot load such file -- emr/common (LoadError) > >>> from > >>> > /usr/share/ruby/vendor_ruby/2.0/rubygems/core_ext/kernel_require.rb:55:in > >>> `require' > >>> from ./setup-drill:39:in `' > >>> > >>> > >>> On Wed, Jun 29, 2016 at 4:04 PM, Paul Mogren > >>> wrote: > >>> > Dave, > > This is a little bit old/outdated but may be better than others you > >> found: > >> > https://github.com/commercehub-oss/watershed/blob/master/watershed/resources/s3/emr/exec/setup_drill > > We haven’t gotten around to updating the overall Watershed project for > >> EMR > 4, which is managed differently, nor more recent versions of Drill. > >> Mostly > because it still meets our needs. But that has discouraged us from > announcing the project, so I’d really like to get it done one of these > >> days. > > Paul Mogren > > > On 6/20/16, 10:29 AM, "David Kincaid" wrote: > > I was wondering if anyone had an up to date bootstrap script that > >> installs > and configures Drill on an AWS EMR cluster. I'm looking for one that > >> will > work with the stock Amazon EMR version and not a MapR EMR cluster. > I've > >> had > too much trouble in the past with the MapR EMR bootstrap scripts > >> themselves > failing. Nowadays I stick to the stock Amazon EMR. I've done some web > searching, but all I can seem to find are pretty old, outdated or in > >> some > cases non-existent scripts. So I know people have tried it in the > past. > >> I'm > very new to Drill, so having something to start with at least would > be a > huge help. > > Thanks, > > Dave > >> >