Re: What is the limit to the number of properties set in the configuration object
they also get sent over the wire with things like job submissions, so can make things slower. in my little grumpy project, https://github.com/steveloughran/grumpy , I actually stuck the groovy scripts into the config files as strings, so they'd be submitted as jobs; the mapper & reducer would simply read the config, parse it as a method under the mapper context, then run it https://github.com/steveloughran/grumpy/blob/master/src/main/groovy/org/apache/hadoop/grumpy/scripted/ScriptedMapper.groovy > On 15 Jun 2015, at 22:35, Colin P. McCabe wrote: > > Much like zombo.com, the only limit is yourself. > > But huge Configuration objects are going to be really inefficient, so > I would look elsewhere for storing lots of data. > > best, > Colin > > On Fri, Jun 12, 2015 at 7:30 PM, Sitaraman Vilayannur > wrote: >> Thanks Allen, what is the total size limit? >> Sitaraman >> >> >> On Fri, Jun 12, 2015 at 10:53 PM, Allen Wittenauer >> wrote: >> >>> >>> On Jun 12, 2015, at 12:37 AM, Sitaraman Vilayannur < >>> vrsitaramanietfli...@gmail.com> wrote: >>> Hi, What is the limit on the number of properties that can be set using set(String s1, String s2) on the Configuration object for hadoop? Is this limit configurable if so what is the maximum that can be set? >>> >>>It's a "total size of the conf" limit, not a "number of" limit. >>> >>>In general, you shouldn't pack it full of stuff as calling >>> Configuration is expensive. Use a side-input/distributed cache file for >>> mass quantities of bits. >>> >>> >>>
Re: What is the limit to the number of properties set in the configuration object
Much like zombo.com, the only limit is yourself. But huge Configuration objects are going to be really inefficient, so I would look elsewhere for storing lots of data. best, Colin On Fri, Jun 12, 2015 at 7:30 PM, Sitaraman Vilayannur wrote: > Thanks Allen, what is the total size limit? > Sitaraman > > > On Fri, Jun 12, 2015 at 10:53 PM, Allen Wittenauer wrote: > >> >> On Jun 12, 2015, at 12:37 AM, Sitaraman Vilayannur < >> vrsitaramanietfli...@gmail.com> wrote: >> >> > Hi, >> > What is the limit on the number of properties that can be set using >> > set(String s1, String s2) on the Configuration object for hadoop? >> > Is this limit configurable if so what is the maximum that can be set? >> >> It's a "total size of the conf" limit, not a "number of" limit. >> >> In general, you shouldn't pack it full of stuff as calling >> Configuration is expensive. Use a side-input/distributed cache file for >> mass quantities of bits. >> >> >>
Re: What is the limit to the number of properties set in the configuration object
Thanks Allen, what is the total size limit? Sitaraman On Fri, Jun 12, 2015 at 10:53 PM, Allen Wittenauer wrote: > > On Jun 12, 2015, at 12:37 AM, Sitaraman Vilayannur < > vrsitaramanietfli...@gmail.com> wrote: > > > Hi, > > What is the limit on the number of properties that can be set using > > set(String s1, String s2) on the Configuration object for hadoop? > > Is this limit configurable if so what is the maximum that can be set? > > It's a "total size of the conf" limit, not a "number of" limit. > > In general, you shouldn't pack it full of stuff as calling > Configuration is expensive. Use a side-input/distributed cache file for > mass quantities of bits. > > >
Re: What is the limit to the number of properties set in the configuration object
On Jun 12, 2015, at 12:37 AM, Sitaraman Vilayannur wrote: > Hi, > What is the limit on the number of properties that can be set using > set(String s1, String s2) on the Configuration object for hadoop? > Is this limit configurable if so what is the maximum that can be set? It's a "total size of the conf" limit, not a "number of" limit. In general, you shouldn't pack it full of stuff as calling Configuration is expensive. Use a side-input/distributed cache file for mass quantities of bits.
What is the limit to the number of properties set in the configuration object
Hi, What is the limit on the number of properties that can be set using set(String s1, String s2) on the Configuration object for hadoop? Is this limit configurable if so what is the maximum that can be set? Thanks Sitaraman
