Re: 100G over 100 km of dark fiber

[Jared Brown]

Turns out I was wrong, again.

There is at least one vendor that offers pluggable 100G QSFP28 optics that can 
reach 120 km with amplification.

Thanks to everybody that reached out! I appreciate all the input and lessons 
learned.


Jared


---

Hello NANOG!

I need to push 100G over 100 km of dark fiber. Since there are no 100G 
pluggable optics with this reach (~25 dB), I have been offered coherent 
transport systems to solve my problem. This is all good and well, except total 
system costs start from high five figures.

So, my question is, do I have any other options?

I can't help noticing that you can break out a 100G QSFP into four 25G QSFPs. 
25G DWDM systems are relatively inexpensive (low five figures), but can you 
make 25G DWDM go 100 km?

I only need the one 100G, so I don't really need a highly scalable DWDM system. 
I can't put anything midspan, or if I could it would cost more than just going 
with a coherent system.


Jared

Re: 100G over 100 km of dark fiber

[Ariën Vijn via NANOG]

Hi Jared,

4x25Gbit/s, 'directly detected' MAY work but it won't be easy at all. As many 
already have suggested, coherent detection will give you much much less 
headaches. 

If you want to go with directly detected for financial reasons than first make 
sure you know the type(s) of fibers and the dispersion you can expect. 
Fiber-operators tend to splice different types of fiber together. Don't just 
assume that it will be all G.652 compatible fiber.

If there is G.655 or even G.653 compatible fiber in your path then it is 
unlikely that you can operate in the O-band (1310nm). 

In that case you need to go with the C-band (1550nm), which is easier to 
amplify than the O-band, because you can use commonly available EDFAs. 

You probably need to compensate the chromatic dispersion in the C-band. For 
that it is again very important to know the fiber type(s) in your path.Because 
a '100km DCM', assumes 100km G.652 fiber. If your path consists of 50km G.652 
and 50km G.655 then you need to compensate for 50km. 

Last but not least, your hardware or optics probably need to be able to do FEC 
to get a bit of a decent BER.

-- Ariën



> On 30 Oct 2020, at 15:19, Jared Brown  wrote:
> 
> Hello NANOG!
> 
> I need to push 100G over 100 km of dark fiber. Since there are no 100G 
> pluggable optics with this reach (~25 dB), I have been offered coherent 
> transport systems to solve my problem. This is all good and well, except 
> total system costs start from high five figures.
> 
> So, my question is, do I have any other options?
> 
> I can't help noticing that you can break out a 100G QSFP into four 25G QSFPs. 
> 25G DWDM systems are relatively inexpensive (low five figures), but can you 
> make 25G DWDM go 100 km?
> 
> I only need the one 100G, so I don't really need a highly scalable DWDM 
> system. I can't put anything midspan, or if I could it would cost more than 
> just going with a coherent system.
> 
> 
> Jared

Re: 100G over 100 km of dark fiber

[Jürgen Jaritsch]

Hi Jared,

Easiest thing would be breaking into sub-100G channels (e.g. 4x 25G) and
using DWDM optics + Amp + Compensation + Pre-Amp to transport it across the
100km.

Same parts for all configs:
*) 1x Amplifier => 1 slot
*) 1x 100km dispersion compensation => separate 1U
*) 1x Pre-Amplifier => 1 slot

Sample config per site:

*) 1U chassis (4 slots)
*) 1x Mgmt card => 1 slot
*) [same parts for all configs]
*) 1x 25G converter (100G client port split into 4x 25G SFP28 line ports) =>
1 slot
*) 1x 8 channel DWDM Mux/Demux => separate 1U
*) 4x 25G SFP28 DWDM 10km
*) 1x 100G QSFP28 SR/LR (this is only for the client port on the converter -
you need a second one for your device)

=> footprint per site ~6200 USD for 1x 100G

The above setup requires 3U of rack space per site and provides 4 more DWDM
channels as spares for further upgrades and the DWDM unit will have an
separate upgrade port, to expand the number of filters (if you do not want
to replace the whole filter with a bigger one).

If you you want to go for a solution which starts with 4x 100G you're on the
13k USD range (per site):

*) 2U chassis (8 slots)
*) 1x Mgmt card => 1 slot
*) [same parts for all configs]
*) 4x 25G converter (100G client port split into 4x 25G SFP28 line ports) =>
4 slots
*) 1x 16 channel DWDM Mux/Demux => separate 1U
*) 16x 25G SFP28 DWDM 10km
*) 4x 100G QSFP28 SR/LR (this is only for the client port on the converter -
you need a second one for your device)

=> footprint per site ~13000 USD for 4x 100G


Even expansion to 12x 100G is more or less "cheap":

*) 5U chassis (16 slots)
*) 1x Mgmt card => 1 slot
*) [same parts for all configs]
*) 12x 25G converter (100G client port split into 4x 25G SFP28 line ports)
=> 12 slots
*) 1x 48 channel DWDM Mux/Demux => separate 1U
*) 48x 25G SFP28 DWDM 10km
*) 12x 100G QSFP28 SR/LR (this is only for the client port on the converter
- you need a second one for your device)

=> footprint per site ~31000 USD for 12x 100G

I typically go for Sintai, FS.com and Edgeoptics components. Regarding the
compensation and AMPs stuff: it's just math, nothing else :).


Best regards
Jürgen

Re: 100G over 100 km of dark fiber

[Lady Benjamin PD Cannon]

Very impressive!  Can you share your fiber type and link-loss?
—L.B.

Lady Benjamin PD Cannon, ASCE
6x7 Networks & 6x7 Telecom, LLC 
CEO 
b...@6by7.net 
"The only fully end-to-end encrypted global telecommunications company in the 
world.”
FCC License KJ6FJJ



> On Oct 30, 2020, at 9:03 AM, Tarko Tikan  wrote:
> 
> hey,
> 
>> If it’s just a single 100G channel needed you could try 100GBASE-ZR4. 
>> Specified for 80km, 30db power budget they could actually reach more the 
>> 80km.
>> Dispersion should also be „no" problem in the 1310nm length. I have to say 
>> that I never tried this on 100km distance without coherent solutions.
> 
> Just to add to my original suggestion, we just did 100G-ZR4 over 30dB link 
> with pre-FEC BER 3.174E-11.
> 
> As OP is asking for a solution for 25dB I don't see any reason why ZR4 would 
> not work and why you would need coherent, amplifiers or any other additional 
> solution except when you are limited by QSFP28 SFF power.
> 
> -- 
> tarko

Re: 100G over 100 km of dark fiber

[Sander Steffann]

Hi,

On 30-10-2020 15:33, Dale W. Carder wrote:


You may also find that 100G PAM4 could work.  There are some vendors that
sell the optic, and an outboard EDFA + DCF pizza box.


We are about to deploy these on a couple of dark fibers:
https://www.solid-optics.com/product/edfamux-multiplexer-amplifier-dispersion-compensation-dwdm-mux-edfa/

They have amplified and dispersion compensated 8x100G to be used with 
PAM4 optics, and a pass-through port to connect existing 1G/10G MUXes to 
(which can have their own amplification if necessary).


They can provide models with different sets of channels of you need that 
(nice when cascading them with existing 1/10G MUXes). IIRC next year 
they can also build in a power meter so you can do remote monitoring.


If you're interested I can let you know how much we like them in a few 
months ;)


Cheers,
Sander

Re: 100G over 100 km of dark fiber

[Jared Brown]

The 100 km leg completes a ring.
 

Jared
 

Sent: Friday, October 30, 2020
From: "Ben Cannon" 
To: "Jared Brown" 
Cc: nanog@nanog.org
Subject: Re: 100G over 100 km of dark fiber

You could break this into 10x 10g coherent lanes, but you’re going to end up 
back close to coherent 100g prices.
 
You’re at the threshold distance where you’re past all the short range tech and 
are seriously pushing it - whereas the 100g coherent tech is just taking off.  
 
How important is this link?
 

Ms. Benjamin PD Cannon, ASCE
6x7 Networks & 6x7 Telecom, LLC 
CEO 
b...@6by7.net[mailto:b...@6by7.net]
"The only fully end-to-end encrypted global telecommunications company in the 
world.”
FCC License KJ6FJJ

 On Oct 30, 2020, at 7:19 AM, Jared Brown  wrote:
 
Hello NANOG!

I need to push 100G over 100 km of dark fiber. Since there are no 100G 
pluggable optics with this reach (~25 dB), I have been offered coherent 
transport systems to solve my problem. This is all good and well, except total 
system costs start from high five figures.

So, my question is, do I have any other options?

I can't help noticing that you can break out a 100G QSFP into four 25G QSFPs. 
25G DWDM systems are relatively inexpensive (low five figures), but can you 
make 25G DWDM go 100 km?

I only need the one 100G, so I don't really need a highly scalable DWDM system. 
I can't put anything midspan, or if I could it would cost more than just going 
with a coherent system.


Jared

Re: 100G over 100 km of dark fiber

[Tarko Tikan]

hey,


If it’s just a single 100G channel needed you could try 100GBASE-ZR4. Specified 
for 80km, 30db power budget they could actually reach more the 80km.
Dispersion should also be „no" problem in the 1310nm length. I have to say that 
I never tried this on 100km distance without coherent solutions.


Just to add to my original suggestion, we just did 100G-ZR4 over 30dB 
link with pre-FEC BER 3.174E-11.


As OP is asking for a solution for 25dB I don't see any reason why ZR4 
would not work and why you would need coherent, amplifiers or any other 
additional solution except when you are limited by QSFP28 SFF power.


--
tarko

Re: 100G over 100 km of dark fiber

[Ben Cannon]

You could break this into 10x 10g coherent lanes, but you’re going to end up 
back close to coherent 100g prices.

You’re at the threshold distance where you’re past all the short range tech and 
are seriously pushing it - whereas the 100g coherent tech is just taking off.  

How important is this link?

Ms. Benjamin PD Cannon, ASCE
6x7 Networks & 6x7 Telecom, LLC 
CEO 
b...@6by7.net
"The only fully end-to-end encrypted global telecommunications company in the 
world.”

FCC License KJ6FJJ



> On Oct 30, 2020, at 7:19 AM, Jared Brown  wrote:
> 
> Hello NANOG!
> 
> I need to push 100G over 100 km of dark fiber. Since there are no 100G 
> pluggable optics with this reach (~25 dB), I have been offered coherent 
> transport systems to solve my problem. This is all good and well, except 
> total system costs start from high five figures.
> 
> So, my question is, do I have any other options?
> 
> I can't help noticing that you can break out a 100G QSFP into four 25G QSFPs. 
> 25G DWDM systems are relatively inexpensive (low five figures), but can you 
> make 25G DWDM go 100 km?
> 
> I only need the one 100G, so I don't really need a highly scalable DWDM 
> system. I can't put anything midspan, or if I could it would cost more than 
> just going with a coherent system.
> 
> 
> Jared

Re: 100G over 100 km of dark fiber

[Nick Hilliard]

Dale W. Carder wrote on 30/10/2020 14:33:

You may also find that 100G PAM4 could work.


not at 100km. This would be outside the dispersion tolerance limits for 
pam4.


Nick

RE: 100G over 100 km of dark fiber

[Brian Turnbow via NANOG]

Hi jared 

as others have pointed out there are lots of options


inphi offers these
https://www.inphi.com/products/colorz/


or use a box like packetlight, here is a Arista solution brief 
https://www.arista.com/assets/data/pdf/Whitepapers/Arista_Packetlight_100G_Extension_Solution.pdf
and if you serch for openline systems there are a few that do smaller systems 
2/4/8 ports that are available
FS even offers this
https://www.fs.com/de-en/specials/100g-fmx-transport-platform-103.html
more expensive than optics  but an alternative and you can stay in low 5 
figures.

cisco has the ncs55a2 with these
https://www.cisco.com/c/en/us/products/collateral/routers/network-convergence-system-5500-series/datasheet-c78-743732.html
that costs like a gazzilion dollars but your company may have great discounts...


HTH

Brian


> -Original Message-
> From: NANOG  On Behalf Of
> Jared Brown
> Sent: Friday, October 30, 2020 3:19 PM
> To: nanog@nanog.org
> Subject: 100G over 100 km of dark fiber
> 
> Hello NANOG!
> 
> I need to push 100G over 100 km of dark fiber. Since there are no 100G
> pluggable optics with this reach (~25 dB), I have been offered coherent
> transport systems to solve my problem. This is all good and well, except total
> system costs start from high five figures.
> 
> So, my question is, do I have any other options?
> 
> I can't help noticing that you can break out a 100G QSFP into four 25G QSFPs.
> 25G DWDM systems are relatively inexpensive (low five figures), but can you
> make 25G DWDM go 100 km?
> 
> I only need the one 100G, so I don't really need a highly scalable DWDM
> system. I can't put anything midspan, or if I could it would cost more than 
> just
> going with a coherent system.
> 
> 
> Jared

Re: 100G over 100 km of dark fiber

[Brandon Martin]

On 10/30/20 10:27 AM, Vincentz Petzholtz wrote:

If it’s just a single 100G channel needed you could try 100GBASE-ZR4. Specified 
for 80km, 30db power budget they could actually reach more the 80km.
Dispersion should also be „no" problem in the 1310nm length. I have to say that 
I never tried this on 100km distance without coherent solutions.


If you end up marginal with the ZR4 at the receive end, you may be able 
to use a silicon optical preamp to make up the margin.


If you are at the noise floor already (ZR4 receivers are pretty good), 
you may just need to dump more power into the line.  I recently came 
across the PDFA (like an EDFA but the fiber is doped with Praseodymium 
instead of Erbium) which operate on the 1310 band and can push upwards 
of 20-23dBm of power out.  They are not cheap (low 5 figures), but they 
are cheaper than coherent and will still let you run on 1310 so you 
don't have chromatic dispersion issues.  With good (ER4 or ZR4) 
receivers good down to -20dBm or so, you've got 30-33dB of link budget 
which should just about make your 100km if it's decent fiber.  They're 
also usable as a pre-amp (different configuration optimized for gain and 
noise figure instead of power, similar to EDFA preamps) and have 
performance potentially better than silicon amps in that situation.


The 1550 PAM4 pizza box EDFA+Mux+DCM that others have mentioned may also 
work and end up even cheaper, but I normally see those for 40-60km or 
"up to" 80km with them really pushing the link budget at that point.


Honestly, I'd be tempted to just suck it up and do a coherent solution, 
though I admit it would probably be at least 2x the cost.  You can 
probably get a 200G carrier, though.

--
Brandon Martin

Re: 100G over 100 km of dark fiber

[Dale W. Carder]

You may also find that 100G PAM4 could work.  There are some vendors that
sell the optic, and an outboard EDFA + DCF pizza box.

Dale

Thus spake Tarko Tikan (ta...@lanparty.ee) on Fri, Oct 30, 2020 at 04:25:58PM 
+0200:
> hey,
> 
> > I need to push 100G over 100 km of dark fiber. Since there are no 100G 
> > pluggable optics with this reach (~25 dB), I have been offered coherent 
> > transport systems to solve my problem. This is all good and well, except 
> > total system costs start from high five figures.
> 
> 100G-ZR4 QSFP28 is on the market and works. Just watch out for power
> limitations, your typical DC switch might not support it but proper stuff
> has no problems providing 6.5W of power.
> 
> -- 
> tarko

Re: 100G over 100 km of dark fiber

[Vincentz Petzholtz]

Hi Jared,

If it’s just a single 100G channel needed you could try 100GBASE-ZR4. Specified 
for 80km, 30db power budget they could actually reach more the 80km.
Dispersion should also be „no" problem in the 1310nm length. I have to say that 
I never tried this on 100km distance without coherent solutions.

Best regards,
Vincentz
PS: https://www.flexoptix.net/en/transceiver/q-161hg-80.html?co10728=100306

> Am 30.10.2020 um 15:19 schrieb Jared Brown :
> 
> Hello NANOG!
> 
> I need to push 100G over 100 km of dark fiber. Since there are no 100G 
> pluggable optics with this reach (~25 dB), I have been offered coherent 
> transport systems to solve my problem. This is all good and well, except 
> total system costs start from high five figures.
> 
> So, my question is, do I have any other options?
> 
> I can't help noticing that you can break out a 100G QSFP into four 25G QSFPs. 
> 25G DWDM systems are relatively inexpensive (low five figures), but can you 
> make 25G DWDM go 100 km?
> 
> I only need the one 100G, so I don't really need a highly scalable DWDM 
> system. I can't put anything midspan, or if I could it would cost more than 
> just going with a coherent system.
> 
> 
> Jared



signature.asc
Description: Message signed with OpenPGP

Re: 100G over 100 km of dark fiber

[Tarko Tikan]

hey,


I need to push 100G over 100 km of dark fiber. Since there are no 100G 
pluggable optics with this reach (~25 dB), I have been offered coherent 
transport systems to solve my problem. This is all good and well, except total 
system costs start from high five figures.


100G-ZR4 QSFP28 is on the market and works. Just watch out for power 
limitations, your typical DC switch might not support it but proper 
stuff has no problems providing 6.5W of power.


--
tarko

100G over 100 km of dark fiber

[Jared Brown]

Hello NANOG!

I need to push 100G over 100 km of dark fiber. Since there are no 100G 
pluggable optics with this reach (~25 dB), I have been offered coherent 
transport systems to solve my problem. This is all good and well, except total 
system costs start from high five figures.

So, my question is, do I have any other options?

I can't help noticing that you can break out a 100G QSFP into four 25G QSFPs. 
25G DWDM systems are relatively inexpensive (low five figures), but can you 
make 25G DWDM go 100 km?

I only need the one 100G, so I don't really need a highly scalable DWDM system. 
I can't put anything midspan, or if I could it would cost more than just going 
with a coherent system.


Jared