Re: Performance issue

[Julien Laurenceau]

Hi
Could you please share more details ?
We can assume it is the same hardware and same OS / JVM.
What about the hadoop configuration ? Are you using Erasure Coding ?

Regards


Le jeu. 12 déc. 2019 à 14:12, HarshaKiran Reddy Boreddy 
a écrit :

> Yes, Even in HDFS(3.1.1), I have seen the performance degradation compared
> with 2.7.2. I have done the DFSIO test following are the details.
>
> Test:
>
> DATA SIZE: 1TB
> No of Files : 1000
> File size : 1GB
>
> *Hadoop-3.1.1::*
>
> *DFSIO Write : 1420.99 sec*
>
> *Hadoop-2.7.2::*
>
> *DFSIO Write : 1297.971 sec*
>
> so,there is degradation compare to 2.7.2..Any pointers to this??
>
> Thanks
> Harshakiran Reddy
>
>
>
>
> From: bilwa st 
> Date: Fri, 22 Nov 2019 at 9:09 PM
> Subject: Performance issue
> To: 
>
>
> Hi,
>
> Tera sort is slow by 9mins(26%) in 3.1.1 compared to 2.7.2
>
> I could see some JIRA open for the same issue ie MAPREDUCE-6735
>
> Has anyone else faced the same issue?
>
> Thanks,
> Bilwa ST
>
>
>
> --
> *Regards*
> Harshakiranreddy.Boreddy
>
>
>

Re: accessing hdfs cluster through ssh tunnel

[Julien Laurenceau]

Hi
Hadoop is designed to avoid proxy as it will act as a bottleneck. Namenodes
are used to obtain a direct socket client / datanodes that is specific to
each job.

Le ven. 13 sept. 2019 à 14:21, Tony S. Wu  a écrit :

> You need connectivity from edge node to the entire cluster, not just
> namenode. Your topology, unfortunately, probably won’t work too well. A
> proper VPN / IPSec tunnel might be a better idea.
>
> On Thu, Sep 12, 2019 at 12:04 AM saurabh pratap singh <
> saurabh.cs...@gmail.com> wrote:
>
>> Hadoop version : 2.8.5
>> I have a hdfs set up in private data center (which is not exposed to
>> internet ) .In the same data center I have another node (gateway
>> node).Purpose of this gateway node is to provide access to hdfs from edge
>> machine (which is present outside of data center) through public internet .
>> To enable this kind of setup I have setup an ssh tunnel from edge machine
>> to name node host and port(9000) through gateway node .
>> something like
>>
>> ssh -N -L ::
>> @ -i   - .
>>
>> When i did hadoop fs -ls hdfs://localhost: it works fine from
>> edge machine but
>> when i executed hadoop fs -put  hdfs://localhost:
>> it fails with following error message.
>>
>> org.apache.hadoop.net.ConnectTimeoutException: 6 millis timeout while
>> waiting for channel to be ready for connect. ch :
>> java.nio.channels.SocketChannel[connection-pending
>> remote=/:50010]
>> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
>> at
>> org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
>> at
>> org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1725)
>> at
>> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1679)
>> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
>>
>>
>> Looks like it is trying to write directly to private ip address of data
>> node .How do i resolve this?
>>
>> Do let me know if some other information is needed .
>>
>> Thanks
>>
>

Re: What is the best way to analyze io latency in hdfs?

[Julien Laurenceau]

Hi,
On Linux you can monitor système call of any process using:

strace -p PIDofHDFSdatanode

It can be very verbose but the information will be there.

Did you try metrics available in ambari or cloudera manager ?

Regards

Le mar. 20 août 2019 à 02:47, Daegyu Han  a écrit :

> Hi all,
>
> I'm currently studying HDFS, and I want to analyze HDFS io latency.
>
> I know that C / C ++ programs can use perf and ftrace under Linux to
> analyze user level and kernel level latency measurements and overhead.
>
> I would like to analyze the read io latency in HDFS to user level (HDFS)
> and system level (kernel I / O stack).
>
> Which way is the best?
>
> Thank you.
>
>
> ᐧ
>

Re: Hadoop HDFS Fault Injection

[Julien Laurenceau]

Hi,
Did you try anarchyape ?
Originally: https://github.com/david78k/anarchyape
my fork to avoid hard coded network interface as "eth0" :
https://github.com/julienlau/anarchyape

Regards,
JL

Le mer. 14 août 2019 à 15:12, Aleksander Buła <
ab370...@students.mimuw.edu.pl> a écrit :

> Hi,
>
> I would like to ask whether the Fault Injection Framework (
> https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/FaultInjectFramework.html)
> is still supported in the Hadoop HDFS?
>
> I think that this documentation was created around *v0.23* and was not
> updated since then. Additionally, I have done some repository digging and
> found out that the ant targets, mentioned in the documentation, were
> deleted in 2012. Right now none of the files in the repository defines
> these targets but there the projects contains multiple *.aj *files -
> therefore I assume they somehow can be used.
>
> Does anyone here know how to compile and run fault injection tests in a
> newer version of Hadoop (exactly* v2.6.0*)?  It would mean a world to me.
>
> Best regards
> Alex
>

Re: [DISCUSS] HDFS roadmap/wish list

[Julien Laurenceau]

Hi,
Thanks for taking this into account.
As you seems to be interested in adapting the "per directory" HDFS
attributes regarding replication factor, I would also suggest to add a
finer granularity to the DFS blocksize.
The idea would be to optimize the blocksize on a per directory basis. So
that an application dealing with very large files could beneficiate from a
very large block size without impacting everyone.
As I am not familiar with the code, I don't know if this kind of feature
would not broke the simplicity of Hadoop.
Maybe this could be done by adding attributes to the hdfs policies.
Maybe it could also be a good idea to make the replication factor an
attribute of the hdfs policies for coherence sake.

Regards
Julien

Le jeu. 13 juin 2019 à 20:41, Wei-Chiu Chuang  a écrit :

> Thank you. I really appreciate your feedback as I don't always know the
> detailed use case for a feature. (For me, it's mostly "hey, this thing is
> broken, fix it")
>
> What are the rest of the community thinks? This is a great opportunity to
> share what you think.
>
> My answers inline:
>
> On Wed, Jun 12, 2019 at 1:12 AM Julien Laurenceau <
> julien.laurenc...@pepitedata.com> wrote:
>
>> Hi,
>>
>> I am not absolutely sure it is not already in a roadmap or supported, but
>> I would appreciate those two features :
>>
>> - First feature : I would also like to be able to use a dedicated
>> directory in HDFS as a /tmp directory leveraging RAMFS for high performing
>> checkpoint of Spark Jobs without using Alluxio or Ignite.
>>
> My current issue is that the RAMFS is only useful with replication factor
>> x1 (in order to avoid network).
>> My default replication factor is x3, but I would need a way to set
>> replication factor x1 on a specific directory (/tmp) for all new writes
>> coming to this directory.
>> Currently if I use "hdfs setrep 1 /tmp" it only works for blocks already
>> written.
>> For example, this could be done by specifying the replication factor at
>> the storage policy level.
>> In my view this would dramatically improve the interest of the
>> Lazy-persist storage policy.
>>
>
> I am told LAZY_PERSIST is never considered a completed feature, and two
> Hadoop distros, CDH and HDP don't support it.
>
> But now that I understand the use case, it looks useful now.
>
>> > From the Doc > Note 1: The Lazy_Persist policy is useful only for
>> single replica blocks. For blocks with more than one replicas, all the
>> replicas will be written to DISK since writing only one of the replicas to
>> RAM_DISK does not improve the overall performance.
>> In the current state of HDFS configuration, I only see the following hack
>> (not tested) to implement such a solution : Configure HDFS replication x1
>> as default configuration and use Erasure Coding RS(6,3) for the main
>> storage by attaching an ec storage policy on all directories except /tmp.
>>
>> hdfs ec -setPolicy -path  [-policy ]
>>
>>
>>
>> - Second feature: a bandwidth throttling dedicated to the re-replication
>> in case of a failed datanode.
>> Something similar to the option dedicated to the balancing algorithm
>> dfs.datanode.balance.bandwidthPerSecbut only for re-replication.
>>
> I am pretty sure I've got people asking about this before a few times.
>
>>
>> Thanks and regards
>> JL
>>
>> Le lun. 10 juin 2019 à 19:08, Wei-Chiu Chuang
>>  a écrit :
>>
>>> Hi!
>>>
>>> I am soliciting feedbacks for HDFS roadmap items and wish list in the
>>> future Hadoop releases. A community meetup
>>> <https://www.meetup.com/Hadoop-Contributors/events/262055924/?rv=ea1_v2&_xtd=gatlbWFpbF9jbGlja9oAJGJiNTE1ODdkLTY0MDAtNDFiZS1iOTU5LTM5ZWYyMDU1N2Q4Nw>
>>> is happening soon, and perhaps we can use this thread to converge on things
>>> we should talk about there.
>>>
>>> I am aware of several major features that merged into trunk, such as
>>> RBF, Consistent Standby Serving Reads, as well as some recent features that
>>> merged into 3.2.0 release (storage policy satisfier).
>>>
>>> What else should we be doing? I have a laundry list of supportability
>>> improvement projects, mostly about improving performance or making
>>> performance diagnostics easier. I can share the list if folks are
>>> interested.
>>>
>>> Are there things we should do to make developer's life easier or things
>>> that would be nice to have for downstream applications? I know Sahil Takiar
>>> made a series of improvements in HDFS for Impala recently, and those
>>> improvements are applicable to other downstreamers such as HBase. Or would
>>> it help if we provide more Hadoop API examples?
>>>
>>

Re: [DISCUSS] HDFS roadmap/wish list

[Julien Laurenceau]

Hi,

I am not absolutely sure it is not already in a roadmap or supported, but I
would appreciate those two features :

- First feature : I would also like to be able to use a dedicated directory
in HDFS as a /tmp directory leveraging RAMFS for high performing checkpoint
of Spark Jobs without using Alluxio or Ignite.
My current issue is that the RAMFS is only useful with replication factor
x1 (in order to avoid network).
My default replication factor is x3, but I would need a way to set
replication factor x1 on a specific directory (/tmp) for all new writes
coming to this directory.
Currently if I use "hdfs setrep 1 /tmp" it only works for blocks already
written.
For example, this could be done by specifying the replication factor at the
storage policy level.
In my view this would dramatically improve the interest of the Lazy-persist
storage policy.

> From the Doc > Note 1: The Lazy_Persist policy is useful only for single
replica blocks. For blocks with more than one replicas, all the replicas
will be written to DISK since writing only one of the replicas to RAM_DISK
does not improve the overall performance.
In the current state of HDFS configuration, I only see the following hack
(not tested) to implement such a solution : Configure HDFS replication x1
as default configuration and use Erasure Coding RS(6,3) for the main
storage by attaching an ec storage policy on all directories except /tmp.

hdfs ec -setPolicy -path  [-policy ]



- Second feature: a bandwidth throttling dedicated to the re-replication in
case of a failed datanode.
Something similar to the option dedicated to the balancing algorithm
dfs.datanode.balance.bandwidthPerSecbut only for re-replication.

Thanks and regards
JL

Le lun. 10 juin 2019 à 19:08, Wei-Chiu Chuang 
a écrit :

> Hi!
>
> I am soliciting feedbacks for HDFS roadmap items and wish list in the
> future Hadoop releases. A community meetup
> 
> is happening soon, and perhaps we can use this thread to converge on things
> we should talk about there.
>
> I am aware of several major features that merged into trunk, such as RBF,
> Consistent Standby Serving Reads, as well as some recent features that
> merged into 3.2.0 release (storage policy satisfier).
>
> What else should we be doing? I have a laundry list of supportability
> improvement projects, mostly about improving performance or making
> performance diagnostics easier. I can share the list if folks are
> interested.
>
> Are there things we should do to make developer's life easier or things
> that would be nice to have for downstream applications? I know Sahil Takiar
> made a series of improvements in HDFS for Impala recently, and those
> improvements are applicable to other downstreamers such as HBase. Or would
> it help if we provide more Hadoop API examples?
>

Deploy HDFS (on openstack VM) over Ceph (bare metal) on same hardware

[Julien Laurenceau]

Hi,

I am designing an elastic rack formed of compute nodes, storage nodes and
openstack control nodes.

I plan to deploy:

   - Ceph on bare metal on storage nodes for ceph-osd and control nodes for
   monitor and metadata.
   - Openstack to orchestrate resources spanning the compute and storage
   nodes.
   - Hadoop worker on VMs instanciated on the storage nodes leveraging ceph
   for storage through cinder volumes.
   - Hadoop master on VMs instanciated on the storage nodes leveraging ceph
   for storage through cinder volumes.
   - Spark, k8s and a bunch of tools on VMs instantiated on the compute
   nodes leveraging ceph for storage through cinder volumes. Workload is
   multi-tenant.

I am really concerned about layering Ceph bare metal under HDFS on the same
physical nodes. This design being mainly driven by lack of resources, it
seems to me that it just cannot reliably withstand load.

I am not yet at the stage of stress tests, but it seems to me that
segregating HDFS bare metal side by side to Ceph-osd bare metal on
different nodes is mandatory.

What would be your pro/cons regarding these designs ?

Regards


PS:
https://superuser.com/questions/1437971/deploy-hdfs-on-openstack-vm-over-ceph-bare-metal-on-same-hardware