Currently, TaskTracker send heatbeat to JobTracker, and JobTracker
will send actions to TaskTracker according TaskTracker's status. I
think you should do some hacking about the heatbeat part of JobTracker
public synchronized HeartbeatResponse heartbeat(TaskTrackerStatus status,
Hi, Thanks you very much for your reply. I want to run my own algorithm for
this part to see if we can achieve better outcome in specific scenario. So
how can I modify it?
Thanks a lot!
Shen
On Thu, Oct 7, 2010 at 6:33 PM, Jeff Zhang wrote:
> I believe it is possible. But what is your purpose
I believe it is possible. But what is your purpose? I believe current
solution is good enough.
On Fri, Oct 8, 2010 at 2:57 AM, Shen LI wrote:
> Hi,
> How can I modify the task assignment strategy in hadoop which is used to
> assign tasks to different worker nodes? (Not the job scheduler)
> Bi
Yes, this relates with the native file system block size and disk
block size, and can reduce the disk fragmentation.
On Fri, Oct 8, 2010 at 2:03 AM, rakesh kothari wrote:
> Is there a reason why block size should be set to some 2^N, for some integer
> N ? Does it help with block defragmentation
On 10/08/2010 02:52 AM, Ted Yu wrote:
> Have you checked
> http://download.oracle.com/javase/6/docs/api/javax/xml/stream/XMLEventReader.html?
Hm, I'm working really often with StAX and I'm using the event reader in
my custom input format to create the records and produce a List of
XMLEvents or mor
Have you checked
http://download.oracle.com/javase/6/docs/api/javax/xml/stream/XMLEventReader.html?
On Thu, Oct 7, 2010 at 5:49 PM, Johannes.Lichtenberger <
johannes.lichtenber...@uni-konstanz.de> wrote:
> On 10/08/2010 02:38 AM, Johannes.Lichtenberger wrote:
> > On 10/08/2010 01:29 AM, Ted Yu wr
On 10/08/2010 02:38 AM, Johannes.Lichtenberger wrote:
> On 10/08/2010 01:29 AM, Ted Yu wrote:
>> http://download.oracle.com/javase/6/docs/api/javax/xml/stream/XMLEventWriter.html
>>
>> You can use an XMLOutputFactory to create an XMLEventWriter, and then use an
>> XMLEventFactory to create events t
If you look at the parameters passed to your mapper/reducer, you should see
something like this:
-Djava.library.path=/opt//hadoop/bin/../lib/native/Linux-amd64-64:/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/job_201010062322_0002/attempt_201010062322_0002_m_00_0/work
On Thu, Oct 7, 20
On 10/08/2010 01:29 AM, Ted Yu wrote:
> http://download.oracle.com/javase/6/docs/api/javax/xml/stream/XMLEventWriter.html
>
> You can use an XMLOutputFactory to create an XMLEventWriter, and then use an
> XMLEventFactory to create events that can then be written to the
> XMLEventWriter.
I just re
Hi there,
My files, each about 40G, content format like:
datadatadatadatadata
datadatadatadatadata
datadatadatadatadata
datadatadatadatadata
datadatadatadatadata
datadatadatadatadata
I don't understand. I want to pass something like
java.library.path="/home/.../libXXX.so" so that every mapper can load that
lib and use the native code.
Could you give me an example?
Thanks!
Yin
On Thu, Oct 7, 2010 at 6:05 PM, Ted Yu wrote:
> Take a look at bin/hadoop:
> hadoop: HADOOP_OPTS=
http://download.oracle.com/javase/6/docs/api/javax/xml/stream/XMLEventWriter.html
You can use an XMLOutputFactory to create an XMLEventWriter, and then use an
XMLEventFactory to create events that can then be written to the
XMLEventWriter.
On Thu, Oct 7, 2010 at 4:05 PM, Johannes.Lichtenberger <
On 10/08/2010 12:01 AM, Ted Yu wrote:
> http://www.ibm.com/developerworks/xml/library/x-stax2.html
I think the problem would be how to serialize XMLEvents or more
precisely I don't know if there's an existing StAX XMLEvent-to-String
class/method.
regards,
Johannes
On 10/08/2010 12:01 AM, Ted Yu wrote:
> http://www.ibm.com/developerworks/xml/library/x-stax2.html
Yes, my approach is to parse a very big XML file (wikipedia revisions)
with StAX in my RecordReader implementation. The key is a timestamp and
the values are Lists, because I don't want to have to
se
Take a look at bin/hadoop:
hadoop: HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"
On Thu, Oct 7, 2010 at 12:04 PM, Yin Lou wrote:
> Hi,
>
> Is there any way to pass system properties, like java.library.path to each
> mapper/reducer?
>
> Thanks,
> Yin
>
http://www.ibm.com/developerworks/xml/library/x-stax2.html
On Thu, Oct 7, 2010 at 2:54 PM, Johannes.Lichtenberger <
johannes.lichtenber...@uni-konstanz.de> wrote:
> On 10/07/2010 05:41 PM, Ted Yu wrote:
> > Since mFormatter.format() returns a String, you don't need to introduce
> > newline.
> > Y
On 10/07/2010 05:41 PM, Ted Yu wrote:
> Since mFormatter.format() returns a String, you don't need to introduce
> newline.
> You can call paramOut.writeUTF() to save the String and call
> paramIn.readUTF() to read it back.
Ok, that's what I did right after replying. My value is a List
or more prec
I wonder if there is a misunderstanding here - the problem is that the
classpath has too many classes on it (and clashes with user classes),
rather than it being a text string which is too long.
I would suggest that the technical discussion of how to fix this goes
onto the JIRA.
Cheers,
Tom
On T
Hi,
Is there any way to pass system properties, like java.library.path to each
mapper/reducer?
Thanks,
Yin
Oops, I meant Writable.
On Thu, Oct 7, 2010 at 6:58 AM, Johannes.Lichtenberger
wrote:
> On 10/07/2010 06:46 AM, Anthony Urso wrote:
>> Hadoop is attempting to cast a Date object to WritableComparable, which
>> Date does not implement, and is causing that exception.
>>
>> Your keys must implement
Hi,
How can I modify the task assignment strategy in hadoop which is used to
assign tasks to different worker nodes? (Not the job scheduler)
Big thanks,
Shen
Is there a reason why block size should be set to some 2^N, for some integer N
? Does it help with block defragmentation etc. ?
Thanks,
-Rakesh
Since mFormatter.format() returns a String, you don't need to introduce
newline.
You can call paramOut.writeUTF() to save the String and call
paramIn.readUTF() to read it back.
The value class doesn't need to implement Comparable.
On Thu, Oct 7, 2010 at 6:58 AM, Johannes.Lichtenberger <
johannes.
On 10/07/2010 06:46 AM, Anthony Urso wrote:
> Hadoop is attempting to cast a Date object to WritableComparable, which
> Date does not implement, and is causing that exception.
>
> Your keys must implement WritableComparable and your values must
> implement Comparable.
The values have to implemen
well, if the issue is a too long classpath, the softlink thingy will give
some room to breath as the total CP length will be much smaller.
A
On Thu, Oct 7, 2010 at 3:43 PM, Henning Blohm wrote:
> So that's actually another issue, right? Besides splitting the classpath
> into those three groups,
So that's actually another issue, right? Besides splitting the classpath
into those three groups, you want the TT to create soft-links on demand
to simplify the computation of classpath string. Is that right?
But it's the TT that actually starts the job VM. Why does it matter what
the string actua
26 matches
Mail list logo
