Re: Fairscheduler - disable "default" pool
Hm, I would like to achieve that if I call poolManager.getPools() I get just pools that I defined in my allocation file.. Harsh I tried your option too but it still shows on the servlet page which means it is present and still persist in poolManager.. Meanwhile I have found that probably I will have to change some code in PoolManager (comment line 144 which is supposed to create the default pool). Thanks both of you for your effort.. On 13 March 2012 21:42, Harsh J wrote: > Merto, > > Are you looking to disable all undefined pools completely, preventing > submissions to it? > For that, "mapred.fairscheduler.allow.undeclared.pools" needs to be > set to false. > > On Wed, Mar 14, 2012 at 1:36 AM, Merto Mertek wrote: > > Thanks for your workaround, but I think that with this you just put a > > constraint on the pool that it will not accept any job. I am doing some > > calculation with weights and do not want the default pool weight to be > > included in the computation.. > > > > > > On 13 March 2012 18:52, Jean-Daniel Cryans wrote: > > > >> We do it here by setting this: > >> > >> 0 > >> > >> So that you _must_ have a pool (that's configured with a different > >> maxRunningJobs) in order to run jobs. > >> > >> Hope this helps, > >> > >> J-D > >> > >> On Tue, Mar 13, 2012 at 10:49 AM, Merto Mertek > >> wrote: > >> > I know that by design all unmarked jobs goes to that pool, however I > am > >> > doing some testing and I am interested if is possible to disable it.. > >> > > >> > Thanks > >> > > > > -- > Harsh J >
Re: Fairscheduler - disable "default" pool
Merto, Are you looking to disable all undefined pools completely, preventing submissions to it? For that, "mapred.fairscheduler.allow.undeclared.pools" needs to be set to false. On Wed, Mar 14, 2012 at 1:36 AM, Merto Mertek wrote: > Thanks for your workaround, but I think that with this you just put a > constraint on the pool that it will not accept any job. I am doing some > calculation with weights and do not want the default pool weight to be > included in the computation.. > > > On 13 March 2012 18:52, Jean-Daniel Cryans wrote: > >> We do it here by setting this: >> >> 0 >> >> So that you _must_ have a pool (that's configured with a different >> maxRunningJobs) in order to run jobs. >> >> Hope this helps, >> >> J-D >> >> On Tue, Mar 13, 2012 at 10:49 AM, Merto Mertek >> wrote: >> > I know that by design all unmarked jobs goes to that pool, however I am >> > doing some testing and I am interested if is possible to disable it.. >> > >> > Thanks >> -- Harsh J
Re: Starting a Hadoop cluster.
Targino, Why are you looking to run Postgres off of HDFS? If you instead want a way to move data between your Hadoop cluster and your existing, external Postgres DB, take a look at Apache Sqoop at http://incubator.apache.org/sqoop/ which is built for exactly data-in/data-out purposes, catering to various DBs. On Wed, Mar 14, 2012 at 12:18 AM, Targino Silveira wrote: > Hi Harsh, > > Another doubt is, HBase is like other database, it's Right ? If yes I can't > use it, because I need to use PostgreSQL. > > When I spoken about tablespace, I sponken about a directory mounted where > postgres will record data, it's should possible with FuseDFS, I don't know > about speed, do you know some case like this. > > My main gol using Hadoop would be to use with Postgres database. > > I thank's for any response. > > Regards, > > 2012/3/13 Harsh J > >> Hi, >> >> On Tue, Mar 13, 2012 at 7:24 AM, Targino Silveira >> wrote: >> > 1º - Can I create a direcory to be used like a tablespace for a database >> ? >> >> Not sure what you mean. Apache Hadoop simply comes with a distributed >> filesystem and a data processing framework that leverages the >> filesystem's features. >> >> If you are looking for something tabular and one that can support >> random querying, you will need to setup Apache HBase on top. >> >> > 2º - Always I need to use commands like this "hadoop dfs -ls /app " to >> work >> > with my cluster ? >> >> Yes this is the default way to interact with the HDFS, but you can >> also setup FuseDFS to have a local fuse-based mount point of your >> HDFS, in case you want one. The process is documented at >> http://wiki.apache.org/hadoop/MountableHDFS >> >> -- >> Harsh J >> > > > > -- > Targino Silveira > +55-85-8626-7297 > www.twitter.com/targinosilveira -- Harsh J
Re: Fairscheduler - disable "default" pool
Thanks for your workaround, but I think that with this you just put a constraint on the pool that it will not accept any job. I am doing some calculation with weights and do not want the default pool weight to be included in the computation.. On 13 March 2012 18:52, Jean-Daniel Cryans wrote: > We do it here by setting this: > > 0 > > So that you _must_ have a pool (that's configured with a different > maxRunningJobs) in order to run jobs. > > Hope this helps, > > J-D > > On Tue, Mar 13, 2012 at 10:49 AM, Merto Mertek > wrote: > > I know that by design all unmarked jobs goes to that pool, however I am > > doing some testing and I am interested if is possible to disable it.. > > > > Thanks >
Re: Starting a Hadoop cluster.
Hi Harsh, Another doubt is, HBase is like other database, it's Right ? If yes I can't use it, because I need to use PostgreSQL. When I spoken about tablespace, I sponken about a directory mounted where postgres will record data, it's should possible with FuseDFS, I don't know about speed, do you know some case like this. My main gol using Hadoop would be to use with Postgres database. I thank's for any response. Regards, 2012/3/13 Harsh J > Hi, > > On Tue, Mar 13, 2012 at 7:24 AM, Targino Silveira > wrote: > > 1º - Can I create a direcory to be used like a tablespace for a database > ? > > Not sure what you mean. Apache Hadoop simply comes with a distributed > filesystem and a data processing framework that leverages the > filesystem's features. > > If you are looking for something tabular and one that can support > random querying, you will need to setup Apache HBase on top. > > > 2º - Always I need to use commands like this "hadoop dfs -ls /app " to > work > > with my cluster ? > > Yes this is the default way to interact with the HDFS, but you can > also setup FuseDFS to have a local fuse-based mount point of your > HDFS, in case you want one. The process is documented at > http://wiki.apache.org/hadoop/MountableHDFS > > -- > Harsh J > -- Targino Silveira +55-85-8626-7297 www.twitter.com/targinosilveira
Re: Fairscheduler - disable "default" pool
We do it here by setting this: 0 So that you _must_ have a pool (that's configured with a different maxRunningJobs) in order to run jobs. Hope this helps, J-D On Tue, Mar 13, 2012 at 10:49 AM, Merto Mertek wrote: > I know that by design all unmarked jobs goes to that pool, however I am > doing some testing and I am interested if is possible to disable it.. > > Thanks
Fairscheduler - disable "default" pool
I know that by design all unmarked jobs goes to that pool, however I am doing some testing and I am interested if is possible to disable it.. Thanks
Permission Request for 'HDFS Users Guide...'
Hello, I've known you website for ages! And I found interesting your publication "HDFS Users Guide" which I googled on http://hadoop.apache.org/common/docs/stable/hdfs_user_guide.html ! I'd love to use it in a project I'm involved with called "Geek Science", so I'm seeking your permission for translation to Macedonian language. "Geek Science" is a freemium-model non-English language orientated startup with collection of scientific articles, personal notes etc. in several languages that is collaboratively edited by volunteers from around the world since 1999. Young and old, students and professors - even your neighbor could be a volunteer member. If you agree, we will credit you for your work in the resulting translation's references by stating that it was based on your work and is used with your permission, and by mentioning the name of my project "Geek Science" back to: http://hadoop.apache.org/common/docs/stable/hdfs_user_guide.html Thank you for your time and patience. I look forward to your response next week. --- Best wishes, John Obri Tirana, Albania (GMT +1) Translation for Education FAQ - http://goo.gl/q9Oq9
SAML based Authentication
Hi All, I have a requirement to integrate a SAML Token based authentication module instead of Kerberos Authentication for authenticating users to Hadoop Servers (NN and JT) . This is for the CLI users and not for the web ui. The server Authentication can still be kerberos based. Does anyone else has this requirement ? Any suggestions on how to go about implementing this ? How much of an effort is this ? Please share your thoughts. Thanks and Regards, Benoy Antony
