Re: ACK behaviour difference LRO/GRO
On Mon, Oct 28, 2013 at 12:34 PM, Markus Stockhausen
stockhau...@collogia.de wrote:
Hello,
about two month we had some problems with IPoIB transfer speeds .
See more http://marc.info/?l=linux-rdmam=137823326109158w=2
After some quite hard test iterations the problem seems to come from the
IPoIB switch from LRO to GRO between kernels 2.6.37 and 2.6.38.
I built a test setup with a 2.6.38 kernel and additionaly compiled a 2.6.37
ib_ipoib module against it. This way I can run a direct comparison
between the old and new module. The major difference between the
two version is inside the ipoib_ib_handle_rx_wc() function:
2.6.37: lro_receive_skb(priv-lro.lro_mgr, skb, NULL);
2.6.38: napi_gro_receive(priv-napi, skb);
As in the last post we use ConnectX cards in datagram mode with a
2044 MTU. We read a file sequentially from a NFS server into /dev/null.
We just want to get the wire speed neglecting hard drives. The
hardware is slightly newer so we get different transfer speeds but
the overall effect should be evident. The server uses a 3.5 kernel and
is not changed during the tests.
With 2.6.37 IPoIB module on the client side and LRO enabled the
speed is 950 MByte/sec. On the NFS server side a tcpdump trace
reads like:
19:51:51.432630 IP 10.10.30.251.nfs 10.10.30.1.781:
Flags [P.], seq 1008434065:1008497161, ack 617432,
win 688, options [nop,nop,TS val 133047292 ecr 429568],
length 63096
19:51:51.432672 IP 10.10.30.1.781 10.10.30.251.nfs:
Flags [.], ack 1008241041, win 24576, options
[nop,nop,TS val 429568 ecr 133047292], length 0
19:51:51.432677 IP 10.10.30.251.nfs 10.10.30.1.781:
Flags [.], seq 1008497161:1008560905, ack 617432,
win 688, options [nop,nop,TS val 133047292 ecr 429568],
length 63744
19:51:51.432725 IP 10.10.30.1.781 10.10.30.251.nfs:
Flags [.], ack 1008304585, win 24576, options
[nop,nop,TS val 429568 ecr 133047292], length 0
19:51:51.432729 IP 10.10.30.251.nfs 10.10.30.1.781:
Flags [.], seq 1008560905:1008624649, ack 617432,
win 688, options [nop,nop,TS val 133047292 ecr 429568],
length 63744
With some slight differences here and there the client sends only
1 ack for about 60k of transferred data. With 2.6.38 module and
onwards (GRO enabled) the speed drops down to 380 MByte/sec
and a different transfer pattern.
19:58:14.631430 IP 10.10.30.251.nfs 10.10.30.1.ircs:
Flags [.], seq 722492293:722502253, ack 442312, win 537,
options [nop,nop,TS val 133143092 ecr 467889], length 9960
19:58:14.631460 IP 10.10.30.1.ircs 10.10.30.251.nfs:
Flags [.], ack 722478181, win 24562, options
[nop,nop,TS val 467889 ecr 133143092], length 0
19:58:14.631485 IP 10.10.30.1.ircs 10.10.30.251.nfs:
Flags [.], ack 722478181, win 24562, options
[nop,nop,TS val 467889 ecr 133143092,nop,nop,sack 1
{722480117:722482333}], length 0
19:58:14.631510 IP 10.10.30.1.ircs 10.10.30.251.nfs:
Flags [.], ack 722488197, win 24562, options [nop,nop,TS
val 467889 ecr 133143092], length 0
19:58:14.631534 IP 10.10.30.1.ircs 10.10.30.251.nfs:
Flags [.], ack 722494229, win 24562, options
[nop,nop,TS val 467889 ecr 133143092], length 0
It seems as if the NFS client acknowledges every 2K packet
separately. I thought that it may come from missing
coalescing parameters and tried a ethtool -C ib0 rx-usecs 5
on both machines but without success.
I'm quite lost now maybe someone can give a tip if I'm
missing something.
Nice work! Look like napi_gro_receive() does not do the work it is
supposed to do ?! My (embedded NFS client) system was on 2.6.38 kernel
but we use ipoib kmod from OFED 1.5.4.1 - so we're still on
lro_receive_skb() path that does not have this issue.
I'll try it out later this week to see what is going on. Mellanox
folks or Roland may have more to say.
-- Wendy
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ACK behaviour difference LRO/GRO
On 28/10/2013 21:34, Markus Stockhausen wrote: After some quite hard test iterations the problem seems to come from the IPoIB switch from LRO to GRO between kernels 2.6.37 and 2.6.38. I built a test setup with a 2.6.38 kernel and additionaly compiled a 2.6.37 ib_ipoib module against it. This way I can run a direct comparison between the old and new module. The major difference between the two version is inside the ipoib_ib_handle_rx_wc() function: 2.6.37: lro_receive_skb(priv-lro.lro_mgr, skb, NULL); 2.6.38: napi_gro_receive(priv-napi, skb); These two commits that went in 3.3 936d7de IPoIB: Stop lying about hard_header_len and use skb-cb to stash LL addresses a0417fa net: Make qdisc_skb_cb upper size bound explicit were supposed to make IPoIB/GRO to work properly, specifically with them, you should see aggregation coming into play I think Yishai Hadas from Mellanox was looking on that too, do we have any insights on the matter? Or. Or. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
AW: ACK behaviour difference LRO/GRO
Von: Or Gerlitz [ogerl...@mellanox.com] Gesendet: Dienstag, 29. Oktober 2013 09:31 An: Markus Stockhausen; linux-rdma@vger.kernel.org; Yishai Hadas Cc: s.wendy.ch...@gmail.com; Erez Shitrit; Saeed Mahameed Betreff: Re: ACK behaviour difference LRO/GRO On 28/10/2013 21:34, Markus Stockhausen wrote: After some quite hard test iterations the problem seems to come from the IPoIB switch from LRO to GRO between kernels 2.6.37 and 2.6.38. I built a test setup with a 2.6.38 kernel and additionaly compiled a 2.6.37 ib_ipoib module against it. This way I can run a direct comparison between the old and new module. The major difference between the two version is inside the ipoib_ib_handle_rx_wc() function: 2.6.37: lro_receive_skb(priv-lro.lro_mgr, skb, NULL); 2.6.38: napi_gro_receive(priv-napi, skb); These two commits that went in 3.3 936d7de IPoIB: Stop lying about hard_header_len and use skb-cb to stash LL addresses a0417fa net: Make qdisc_skb_cb upper size bound explicit were supposed to make IPoIB/GRO to work properly, specifically with them, you should see aggregation coming into play I think Yishai Hadas from Mellanox was looking on that too, do we have any insights on the matter? Or. At least for the 2.6.38 that sounds clear. My initial post was about 3.5. and 3.10 test kernels that showed the missing aggregation. So I'm still a bit away from a solution. I will try to get the test machine back to 3.10/3.11 to validate it once again. Just to be on the right way: What are the basics to get GRO working with a ConnectX (not 2 or 3) card in 2044 MTU datagram mode? - enable GRO with ethtool. - Activate Coalescing with ethtool? If yes how? Thanks for the help. Markus Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Ãber das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497
Re: AW: ACK behaviour difference LRO/GRO
On 29/10/2013 13:10, Markus Stockhausen wrote: Just to be on the right way: What are the basics to get GRO working with a ConnectX (not 2 or 3) card in 2044 MTU datagram mode? - enable GRO with ethtool. - Activate Coalescing with ethtool? If yes how? GRO is SW element of the network stack, so the HCA doesn't matter, you need IPoIB to support that correct and enable it with ethtool -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: AW: ACK behaviour difference LRO/GRO
בתאריך 10/29/2013 1:43 PM, ציטוט Or Gerlitz: On 29/10/2013 13:10, Markus Stockhausen wrote: Just to be on the right way: What are the basics to get GRO working with a ConnectX (not 2 or 3) card in 2044 MTU datagram mode? - enable GRO with ethtool. - Activate Coalescing with ethtool? If yes how? GRO is SW element of the network stack, so the HCA doesn't matter, you need IPoIB to support that correct and enable it with ethtool -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html In addition to what Or just wrote, GRO currently doesn't work on ipoib interfaces, that according to bad handling mac address that are not 6 bytes (we have plans to fix that in the near future), that is the reason you don't see 64k packets on tcpdump (like you see in LRO). you have few options: 1. use old kernels and enable LRO (which works fine with ipoib) 2. use CM mode instead of UD, that probably will improve the performance. Thanks, Erez -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: AW: ACK behaviour difference LRO/GRO
On 29/10/2013 14:55, Erez Shitrit wrote: In addition to what Or just wrote, GRO currently doesn't work on ipoib interfaces, that according to bad handling mac address that are not 6 bytes (we have plans to fix that in the near future), that is the reason you don't see 64k packets on tcpdump (like you see in LRO). I just checked with net-next which is 3.12-rc6+ and there IS aggregationfor datagram mode 15:56:40.983883 IP 192.168.20.18.55714 192.168.20.17.40861: Flags [.], seq 1801688305:1801692289, ack 1, win 220, options [nop,nop,TS val 44014459 ecr 305403520], length 3984 15:56:40.983942 IP 192.168.20.18.55714 192.168.20.17.40861: Flags [.], seq 1801692289:1801756033, ack 1, win 220, options [nop,nop,TS val 44014459 ecr 305403520], length 63744 15:56:40.984027 IP 192.168.20.18.55714 192.168.20.17.40861: Flags [.], seq 1801756033:1801819777, ack 1, win 220, options [nop,nop,TS val 44014459 ecr 305403520], length 63744 15:56:40.984079 IP 192.168.20.17.40861 192.168.20.18.55714: Flags [.], ack 1801688305, win 1544, options [nop,nop,TS val 305403520 ecr 44014459], length 0 15:56:40.984104 IP 192.168.20.18.55714 192.168.20.17.40861: Flags [.], seq 1801819777:1801823649, ack 1, win 220, options [nop,nop,TS val 44014459 ecr 305403520], length 3872 15:56:40.984159 IP 192.168.20.18.55714 192.168.20.17.40861: Flags [.], seq 1801823649:1801883521, ack 1, win 220, options [nop,nop,TS val 44014459 ecr 305403520], length 59872 15:56:40.984214 IP 192.168.20.17.40861 192.168.20.18.55714: Flags [.], ack 1801819777, win 1009, options [nop,nop,TS val 305403520 ecr 44014459], length 0 15:56:40.984241 IP 192.168.20.18.55714 192.168.20.17.40861: Flags [.], seq 1801883521:1801887393, ack 1, win 220, options [nop,nop,TS val 44014459 ecr 305403520], length 3872 -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
AW: AW: ACK behaviour difference LRO/GRO
Von: Or Gerlitz [ogerl...@mellanox.com] Gesendet: Dienstag, 29. Oktober 2013 14:58 An: Erez Shitrit Cc: Markus Stockhausen; linux-rdma@vger.kernel.org; Yishai Hadas; s.wendy.ch...@gmail.com; Erez Shitrit; Saeed Mahameed Betreff: Re: AW: ACK behaviour difference LRO/GRO On 29/10/2013 14:55, Erez Shitrit wrote: In addition to what Or just wrote, GRO currently doesn't work on ipoib interfaces, that according to bad handling mac address that are not 6 bytes (we have plans to fix that in the near future), that is the reason you don't see 64k packets on tcpdump (like you see in LRO). I just checked with net-next which is 3.12-rc6+ and there IS aggregationfor datagram mode 15:56:40.983883 IP 192.168.20.18.55714 192.168.20.17.40861: Flags [.], seq 1801688305:1801692289, ack 1, win 220, options [nop,nop,TS val 44014459 ecr 305403520], length 3984 15:56:40.983942 IP 192.168.20.18.55714 192.168.20.17.40861: Flags [.], seq 1801692289:1801756033, ack 1, win 220, options [nop,nop,TS val 44014459 ecr 305403520], length 63744 15:56:40.984027 IP 192.168.20.18.55714 192.168.20.17.40861: Flags [.], seq 1801756033:1801819777, ack 1, win 220, options [nop,nop,TS val 44014459 ecr 305403520], length 63744 15:56:40.984079 IP 192.168.20.17.40861 192.168.20.18.55714: Flags [.], ack 1801688305, win 1544, options [nop,nop,TS val 305403520 ecr 44014459], length 0 15:56:40.984104 IP 192.168.20.18.55714 192.168.20.17.40861: Flags [.], seq 1801819777:1801823649, ack 1, win 220, options [nop,nop,TS val 44014459 ecr 305403520], length 3872 15:56:40.984159 IP 192.168.20.18.55714 192.168.20.17.40861: Flags [.], seq 1801823649:1801883521, ack 1, win 220, options [nop,nop,TS val 44014459 ecr 305403520], length 59872 15:56:40.984214 IP 192.168.20.17.40861 192.168.20.18.55714: Flags [.], ack 1801819777, win 1009, options [nop,nop,TS val 305403520 ecr 44014459], length 0 15:56:40.984241 IP 192.168.20.18.55714 192.168.20.17.40861: Flags [.], seq 1801883521:1801887393, ack 1, win 220, options [nop,nop,TS val 44014459 ecr 305403520], length 3872 Thanks to both of you for that clarification. Nevertheless this info seems a little contradictionary. Should I exepect GRO to work on Mellanox IB cards with Linux 3.12 in general? Or is this only an effect because you have test cards with good mac addresses? Markus Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Ãber das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497
Re: AW: AW: ACK behaviour difference LRO/GRO
On 29/10/2013 17:54, Markus Stockhausen wrote: Should I exepect GRO to work on Mellanox IB cards with Linux 3.12 in general? YES -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: AW: AW: ACK behaviour difference LRO/GRO
Von: Or Gerlitz [ogerl...@mellanox.com] Gesendet: Dienstag, 29. Oktober 2013 16:55 An: Markus Stockhausen; Erez Shitrit Cc: linux-rdma@vger.kernel.org; Yishai Hadas; s.wendy.ch...@gmail.com; Saeed Mahameed Betreff: Re: AW: AW: ACK behaviour difference LRO/GRO On 29/10/2013 17:54, Markus Stockhausen wrote: Should I exepect GRO to work on Mellanox IB cards with Linux 3.12 in general? YES I'm so sorry to tell you that a jump on the 3.12 rc6 bandwagon did not gave a better picture. To sum up the most recent situation: - NFS server based on a 3.5 kernel - Server has an older Xeon L5420 - NFS client based on a 3.12 rc6 kernel - NFS version 3/4 does not matter - Client has a newer Core i7 - Both ends use datagram mode with 2K MTU - Client reads data from NFS (RAM) with dd into null device - tcpdump is running on the server side. From the dump I can see most of the time about 20 incoming ack packets for one sent out data packet of size 64k. Statistics from /proc/interrupts on the server side give a result that fits the picture. ~87 interrupts for 4GB transferred data. That is round about one interrupt per 4,5K. Not very scientific but at least an idea of what is going on. With the mac address explanation of Erez in mind I would confirm that there is still room to improve GRO even in newest kernels. Markus Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Ãber das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497
ACK behaviour difference LRO/GRO
Hello,
about two month we had some problems with IPoIB transfer speeds .
See more http://marc.info/?l=linux-rdmam=137823326109158w=2
After some quite hard test iterations the problem seems to come from the
IPoIB switch from LRO to GRO between kernels 2.6.37 and 2.6.38.
I built a test setup with a 2.6.38 kernel and additionaly compiled a 2.6.37
ib_ipoib module against it. This way I can run a direct comparison
between the old and new module. The major difference between the
two version is inside the ipoib_ib_handle_rx_wc() function:
2.6.37: lro_receive_skb(priv-lro.lro_mgr, skb, NULL);
2.6.38: napi_gro_receive(priv-napi, skb);
As in the last post we use ConnectX cards in datagram mode with a
2044 MTU. We read a file sequentially from a NFS server into /dev/null.
We just want to get the wire speed neglecting hard drives. The
hardware is slightly newer so we get different transfer speeds but
the overall effect should be evident. The server uses a 3.5 kernel and
is not changed during the tests.
With 2.6.37 IPoIB module on the client side and LRO enabled the
speed is 950 MByte/sec. On the NFS server side a tcpdump trace
reads like:
19:51:51.432630 IP 10.10.30.251.nfs 10.10.30.1.781:
Flags [P.], seq 1008434065:1008497161, ack 617432,
win 688, options [nop,nop,TS val 133047292 ecr 429568],
length 63096
19:51:51.432672 IP 10.10.30.1.781 10.10.30.251.nfs:
Flags [.], ack 1008241041, win 24576, options
[nop,nop,TS val 429568 ecr 133047292], length 0
19:51:51.432677 IP 10.10.30.251.nfs 10.10.30.1.781:
Flags [.], seq 1008497161:1008560905, ack 617432,
win 688, options [nop,nop,TS val 133047292 ecr 429568],
length 63744
19:51:51.432725 IP 10.10.30.1.781 10.10.30.251.nfs:
Flags [.], ack 1008304585, win 24576, options
[nop,nop,TS val 429568 ecr 133047292], length 0
19:51:51.432729 IP 10.10.30.251.nfs 10.10.30.1.781:
Flags [.], seq 1008560905:1008624649, ack 617432,
win 688, options [nop,nop,TS val 133047292 ecr 429568],
length 63744
With some slight differences here and there the client sends only
1 ack for about 60k of transferred data. With 2.6.38 module and
onwards (GRO enabled) the speed drops down to 380 MByte/sec
and a different transfer pattern.
19:58:14.631430 IP 10.10.30.251.nfs 10.10.30.1.ircs:
Flags [.], seq 722492293:722502253, ack 442312, win 537,
options [nop,nop,TS val 133143092 ecr 467889], length 9960
19:58:14.631460 IP 10.10.30.1.ircs 10.10.30.251.nfs:
Flags [.], ack 722478181, win 24562, options
[nop,nop,TS val 467889 ecr 133143092], length 0
19:58:14.631485 IP 10.10.30.1.ircs 10.10.30.251.nfs:
Flags [.], ack 722478181, win 24562, options
[nop,nop,TS val 467889 ecr 133143092,nop,nop,sack 1
{722480117:722482333}], length 0
19:58:14.631510 IP 10.10.30.1.ircs 10.10.30.251.nfs:
Flags [.], ack 722488197, win 24562, options [nop,nop,TS
val 467889 ecr 133143092], length 0
19:58:14.631534 IP 10.10.30.1.ircs 10.10.30.251.nfs:
Flags [.], ack 722494229, win 24562, options
[nop,nop,TS val 467889 ecr 133143092], length 0
It seems as if the NFS client acknowledges every 2K packet
separately. I thought that it may come from missing
coalescing parameters and tried a ethtool -C ib0 rx-usecs 5
on both machines but without success.
I'm quite lost now maybe someone can give a tip if I'm
missing something.
Best regards.
Markus
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.
Ãber das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
Vorstand:
Kadir Akin
Dr. Michael Höhnerbach
Vorsitzender des Aufsichtsrates:
Hans Kristian Langva
Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497
This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
executive board:
Kadir Akin
Dr. Michael Höhnerbach
President of the supervisory board:
Hans Kristian Langva
Registry office: district court Cologne
Register number: HRB 52 497
