[casper] Timing distribution over fiber
Hi CASPERites, For HERA, we're looking at distributing timing signals (PPS & 10Mhz ref or 500 MHz clock) over O(100m) fibers to various digitization nodes. I figure some folks in CASPERland have experience with this kind of system? Did you use custom RF-over-fiber kit, or off-the-shelf PPS/10MHz solutions? Any words of wisdom/caution to share? Any responses much appreciated! Jack
Re: [casper] Timing distribution over fiber
> Hi CASPERites, > > For HERA, we're looking at distributing timing signals (PPS & 10Mhz ref or > 500 MHz clock) over O(100m) fibers to various digitization nodes. > I figure some folks in CASPERland have experience with this kind of > system? > Did you use custom RF-over-fiber kit, or off-the-shelf PPS/10MHz > solutions? > Any words of wisdom/caution to share? > > Any responses much appreciated! We have several different schemes for the different signals. Are you planning for one fiber per signal per node? or one fiber with the signals multiplexed on them? If the signals are one per signal, you can use some off-the-shelf solutions, but they are kind of pricey, and if you have a lot of nodes to supply, it might be worth working on something custom. We have used Math Associates stuff for this kind of work. Math Associates is now litelink, and they tout the affordability of their stuff, so maybe it's reasonable... On the 10 MHz, we send the 10 MHz reference over fiber, and at the far end use a crystal oscillator locked to the reference to clean up the noise from the fiber electronics. This is essential for interferometry, but maybe not for single-dish use. John > > Jack >
Re: [casper] Timing distribution over fiber
Hi John, Thanks for the info. I'll add Litelink to my list of suppliers to investigate. We have no particular urge to multiplex the signals on to the fiber unless there's a particularly neat/cheap solution to do that. There's no great appetite to go custom. We've got about ~30 nodes, and my first stab at getting an off-the-shelf solution turned up at a few k$ / node, not including any cleanup electronics. Thanks again, Jack On Mon, 4 May 2015 at 19:25 John Ford wrote: > > Hi CASPERites, > > > > For HERA, we're looking at distributing timing signals (PPS & 10Mhz ref > or > > 500 MHz clock) over O(100m) fibers to various digitization nodes. > > I figure some folks in CASPERland have experience with this kind of > > system? > > Did you use custom RF-over-fiber kit, or off-the-shelf PPS/10MHz > > solutions? > > Any words of wisdom/caution to share? > > > > Any responses much appreciated! > > We have several different schemes for the different signals. Are you > planning for one fiber per signal per node? or one fiber with the signals > multiplexed on them? > > If the signals are one per signal, you can use some off-the-shelf > solutions, but they are kind of pricey, and if you have a lot of nodes to > supply, it might be worth working on something custom. We have used Math > Associates stuff for this kind of work. Math Associates is now litelink, > and they tout the affordability of their stuff, so maybe it's > reasonable... > > > On the 10 MHz, we send the 10 MHz reference over fiber, and at the far end > use a crystal oscillator locked to the reference to clean up the noise > from the fiber electronics. This is essential for interferometry, but > maybe not for single-dish use. > > John > > > > > Jack > > > > >
Re: [casper] Timing distribution over fiber
Hi Jack and John, I wanted to add an input here….. I am working on a 10 MHz GPS slaved reference for my personal use. I am working with a Analog Devices AD9548 Evaluation board (~$250) , GPS with 1 PPS, and a ovenized 10 MHz osc. I also plan to distribute this clock and have considered the Avago fiber product line. One of the older generation Avago fiber parts should work fine for <$25 per channel. With careful control of lengths and delays it should be possible to maintain good phasing between channels. The analog devices chip is <$50 so a custom solution should be <$500/reference but with considerable development time. Bob Stricklin On May 4, 2015, at 10:02 PM, Jack Hickish mailto:jackhick...@gmail.com>> wrote: Hi John, Thanks for the info. I'll add Litelink to my list of suppliers to investigate. We have no particular urge to multiplex the signals on to the fiber unless there's a particularly neat/cheap solution to do that. There's no great appetite to go custom. We've got about ~30 nodes, and my first stab at getting an off-the-shelf solution turned up at a few k$ / node, not including any cleanup electronics. Thanks again, Jack On Mon, 4 May 2015 at 19:25 John Ford mailto:jf...@nrao.edu>> wrote: > Hi CASPERites, > > For HERA, we're looking at distributing timing signals (PPS & 10Mhz ref or > 500 MHz clock) over O(100m) fibers to various digitization nodes. > I figure some folks in CASPERland have experience with this kind of > system? > Did you use custom RF-over-fiber kit, or off-the-shelf PPS/10MHz > solutions? > Any words of wisdom/caution to share? > > Any responses much appreciated! We have several different schemes for the different signals. Are you planning for one fiber per signal per node? or one fiber with the signals multiplexed on them? If the signals are one per signal, you can use some off-the-shelf solutions, but they are kind of pricey, and if you have a lot of nodes to supply, it might be worth working on something custom. We have used Math Associates stuff for this kind of work. Math Associates is now litelink, and they tout the affordability of their stuff, so maybe it's reasonable... On the 10 MHz, we send the 10 MHz reference over fiber, and at the far end use a crystal oscillator locked to the reference to clean up the noise from the fiber electronics. This is essential for interferometry, but maybe not for single-dish use. John > > Jack >
Re: [casper] Timing distribution over fiber
On the far end of the concept spectrum, have you considered distributing time over your existing ethernet network with IEEE-1588, and using this to discipline local ovenised 10MHz oscs at each antenna? I'm cc'ing Johan Burger, who heads up our Timing and Frequency Reference subsystem, who might be able to offer some additional insight. I know they've tried a few different lasers and detectors, with varying levels of success. Jason Manley CBF Manager SKA-SA Cell: +27 82 662 7726 Work: +27 21 506 7300 On 05 May 2015, at 5:18, Bob Stricklin wrote: > Hi Jack and John, > > I wanted to add an input here….. > > I am working on a 10 MHz GPS slaved reference for my personal use. I am > working with a Analog Devices AD9548 Evaluation board (~$250) , GPS with 1 > PPS, and a ovenized 10 MHz osc. I also plan to distribute this clock and have > considered the Avago fiber product line. One of the older generation Avago > fiber parts should work fine for <$25 per channel. With careful control of > lengths and delays it should be possible to maintain good phasing between > channels. The analog devices chip is <$50 so a custom solution should be > <$500/reference but with considerable development time. > > Bob Stricklin > > >> On May 4, 2015, at 10:02 PM, Jack Hickish wrote: >> >> Hi John, >> >> Thanks for the info. I'll add Litelink to my list of suppliers to >> investigate. >> We have no particular urge to multiplex the signals on to the fiber unless >> there's a particularly neat/cheap solution to do that. There's no great >> appetite to go custom. We've got about ~30 nodes, and my first stab at >> getting an off-the-shelf solution turned up at a few k$ / node, not >> including any cleanup electronics. >> >> Thanks again, >> >> Jack >> >> On Mon, 4 May 2015 at 19:25 John Ford wrote: >> > Hi CASPERites, >> > >> > For HERA, we're looking at distributing timing signals (PPS & 10Mhz ref or >> > 500 MHz clock) over O(100m) fibers to various digitization nodes. >> > I figure some folks in CASPERland have experience with this kind of >> > system? >> > Did you use custom RF-over-fiber kit, or off-the-shelf PPS/10MHz >> > solutions? >> > Any words of wisdom/caution to share? >> > >> > Any responses much appreciated! >> >> We have several different schemes for the different signals. Are you >> planning for one fiber per signal per node? or one fiber with the signals >> multiplexed on them? >> >> If the signals are one per signal, you can use some off-the-shelf >> solutions, but they are kind of pricey, and if you have a lot of nodes to >> supply, it might be worth working on something custom. We have used Math >> Associates stuff for this kind of work. Math Associates is now litelink, >> and they tout the affordability of their stuff, so maybe it's >> reasonable... >> >> >> On the 10 MHz, we send the 10 MHz reference over fiber, and at the far end >> use a crystal oscillator locked to the reference to clean up the noise >> from the fiber electronics. This is essential for interferometry, but >> maybe not for single-dish use. >> >> John >> >> > >> > Jack >> > >> >> >
Re: [casper] Timing distribution over fiber
Hi, Jason, I have a great deal of curiosity about IEEE-1588, but I've always wondered about the precision/stability that's attainable. Compared with multiple sample clocks, correlating signals sampled with one common clock seems far more forgiving vis a vis clock frequency/phase stability. If you or John could point me to any information about this, please do! Thanks, Dave On May 4, 2015, at 11:44 PM, Jason Manley wrote: > On the far end of the concept spectrum, have you considered distributing time > over your existing ethernet network with IEEE-1588, and using this to > discipline local ovenised 10MHz oscs at each antenna? > > I'm cc'ing Johan Burger, who heads up our Timing and Frequency Reference > subsystem, who might be able to offer some additional insight. I know they've > tried a few different lasers and detectors, with varying levels of success. > > Jason Manley > CBF Manager > SKA-SA > > Cell: +27 82 662 7726 > Work: +27 21 506 7300 > > On 05 May 2015, at 5:18, Bob Stricklin wrote: > >> Hi Jack and John, >> >> I wanted to add an input here….. >> >> I am working on a 10 MHz GPS slaved reference for my personal use. I am >> working with a Analog Devices AD9548 Evaluation board (~$250) , GPS with 1 >> PPS, and a ovenized 10 MHz osc. I also plan to distribute this clock and >> have considered the Avago fiber product line. One of the older generation >> Avago fiber parts should work fine for <$25 per channel. With careful >> control of lengths and delays it should be possible to maintain good phasing >> between channels. The analog devices chip is <$50 so a custom solution >> should be <$500/reference but with considerable development time. >> >> Bob Stricklin >> >> >>> On May 4, 2015, at 10:02 PM, Jack Hickish wrote: >>> >>> Hi John, >>> >>> Thanks for the info. I'll add Litelink to my list of suppliers to >>> investigate. >>> We have no particular urge to multiplex the signals on to the fiber unless >>> there's a particularly neat/cheap solution to do that. There's no great >>> appetite to go custom. We've got about ~30 nodes, and my first stab at >>> getting an off-the-shelf solution turned up at a few k$ / node, not >>> including any cleanup electronics. >>> >>> Thanks again, >>> >>> Jack >>> >>> On Mon, 4 May 2015 at 19:25 John Ford wrote: Hi CASPERites, For HERA, we're looking at distributing timing signals (PPS & 10Mhz ref or 500 MHz clock) over O(100m) fibers to various digitization nodes. I figure some folks in CASPERland have experience with this kind of system? Did you use custom RF-over-fiber kit, or off-the-shelf PPS/10MHz solutions? Any words of wisdom/caution to share? Any responses much appreciated! >>> >>> We have several different schemes for the different signals. Are you >>> planning for one fiber per signal per node? or one fiber with the signals >>> multiplexed on them? >>> >>> If the signals are one per signal, you can use some off-the-shelf >>> solutions, but they are kind of pricey, and if you have a lot of nodes to >>> supply, it might be worth working on something custom. We have used Math >>> Associates stuff for this kind of work. Math Associates is now litelink, >>> and they tout the affordability of their stuff, so maybe it's >>> reasonable... >>> >>> >>> On the 10 MHz, we send the 10 MHz reference over fiber, and at the far end >>> use a crystal oscillator locked to the reference to clean up the noise >>> from the fiber electronics. This is essential for interferometry, but >>> maybe not for single-dish use. >>> >>> John >>> Jack >>> >>> >> > >
Re: [casper] Timing distribution over fiber
This is not directed at anyone in particular, but it seems appropriate to mention the White Rabbit project (can’t see it mentioned in the CASPER email archive): http://www.ohwr.org/projects/white-rabbit http://en.wikipedia.org/wiki/The_White_Rabbit_Project It is based on IEEE-1588, but improves on those specs at the expense of requiring some custom switch hardware, which is in turn open source. In particular, the 10 MHz provided at each station is phase locked, I believe. Jack may consider looking into it. It works over a few km, so appropriate for HERA if it works in other respects. Jonathan > On May 5, 2015, at 11:52 AM, David MacMahon wrote: > > Hi, Jason, > > I have a great deal of curiosity about IEEE-1588, but I've always wondered > about the precision/stability that's attainable. Compared with multiple > sample clocks, correlating signals sampled with one common clock seems far > more forgiving vis a vis clock frequency/phase stability. If you or John > could point me to any information about this, please do! > > Thanks, > Dave > > On May 4, 2015, at 11:44 PM, Jason Manley wrote: > >> On the far end of the concept spectrum, have you considered distributing >> time over your existing ethernet network with IEEE-1588, and using this to >> discipline local ovenised 10MHz oscs at each antenna? >> >> I'm cc'ing Johan Burger, who heads up our Timing and Frequency Reference >> subsystem, who might be able to offer some additional insight. I know >> they've tried a few different lasers and detectors, with varying levels of >> success. >> >> Jason Manley >> CBF Manager >> SKA-SA >> >> Cell: +27 82 662 7726 >> Work: +27 21 506 7300 >> >> On 05 May 2015, at 5:18, Bob Stricklin wrote: >> >>> Hi Jack and John, >>> >>> I wanted to add an input here….. >>> >>> I am working on a 10 MHz GPS slaved reference for my personal use. I am >>> working with a Analog Devices AD9548 Evaluation board (~$250) , GPS with 1 >>> PPS, and a ovenized 10 MHz osc. I also plan to distribute this clock and >>> have considered the Avago fiber product line. One of the older generation >>> Avago fiber parts should work fine for <$25 per channel. With careful >>> control of lengths and delays it should be possible to maintain good >>> phasing between channels. The analog devices chip is <$50 so a custom >>> solution should be <$500/reference but with considerable development time. >>> >>> Bob Stricklin >>> >>> On May 4, 2015, at 10:02 PM, Jack Hickish wrote: Hi John, Thanks for the info. I'll add Litelink to my list of suppliers to investigate. We have no particular urge to multiplex the signals on to the fiber unless there's a particularly neat/cheap solution to do that. There's no great appetite to go custom. We've got about ~30 nodes, and my first stab at getting an off-the-shelf solution turned up at a few k$ / node, not including any cleanup electronics. Thanks again, Jack On Mon, 4 May 2015 at 19:25 John Ford wrote: > Hi CASPERites, > > For HERA, we're looking at distributing timing signals (PPS & 10Mhz ref or > 500 MHz clock) over O(100m) fibers to various digitization nodes. > I figure some folks in CASPERland have experience with this kind of > system? > Did you use custom RF-over-fiber kit, or off-the-shelf PPS/10MHz > solutions? > Any words of wisdom/caution to share? > > Any responses much appreciated! We have several different schemes for the different signals. Are you planning for one fiber per signal per node? or one fiber with the signals multiplexed on them? If the signals are one per signal, you can use some off-the-shelf solutions, but they are kind of pricey, and if you have a lot of nodes to supply, it might be worth working on something custom. We have used Math Associates stuff for this kind of work. Math Associates is now litelink, and they tout the affordability of their stuff, so maybe it's reasonable... On the 10 MHz, we send the 10 MHz reference over fiber, and at the far end use a crystal oscillator locked to the reference to clean up the noise from the fiber electronics. This is essential for interferometry, but maybe not for single-dish use. John > > Jack > >>> >> >> > >
Re: [casper] Timing distribution over fiber
hi dave, i also think distributing clock and 1 PPS is simpler than IEEE1588. some of the IEEE1588 and white rabbit experts are here at berkeley. see for example: http://chess.eecs.berkeley.edu/pubs/881/dreams.pdf my limited understanding is that 1588 phase locks the bit clocks of routers and switches together. 1588 routers and switches have SMA connectors on them so you can use external maser/rubidium/GPS references. you can achieve spectacular accuracy and stabilitity with white rabbit if you employ really good oscillators at each node, i think white rabbit can acheive 70 picosecond RMS time transfer. best, dan On Tue, May 5, 2015 at 8:52 AM, David MacMahon wrote: > Hi, Jason, > > I have a great deal of curiosity about IEEE-1588, but I've always wondered > about the precision/stability that's attainable. Compared with multiple > sample clocks, correlating signals sampled with one common clock seems far > more forgiving vis a vis clock frequency/phase stability. If you or John > could point me to any information about this, please do! > > Thanks, > Dave > > On May 4, 2015, at 11:44 PM, Jason Manley wrote: > > > On the far end of the concept spectrum, have you considered distributing > time over your existing ethernet network with IEEE-1588, and using this to > discipline local ovenised 10MHz oscs at each antenna? > > > > I'm cc'ing Johan Burger, who heads up our Timing and Frequency Reference > subsystem, who might be able to offer some additional insight. I know > they've tried a few different lasers and detectors, with varying levels of > success. > > > > Jason Manley > > CBF Manager > > SKA-SA > > > > Cell: +27 82 662 7726 > > Work: +27 21 506 7300 > > > > On 05 May 2015, at 5:18, Bob Stricklin wrote: > > > >> Hi Jack and John, > >> > >> I wanted to add an input here….. > >> > >> I am working on a 10 MHz GPS slaved reference for my personal use. I am > working with a Analog Devices AD9548 Evaluation board (~$250) , GPS with 1 > PPS, and a ovenized 10 MHz osc. I also plan to distribute this clock and > have considered the Avago fiber product line. One of the older generation > Avago fiber parts should work fine for <$25 per channel. With careful > control of lengths and delays it should be possible to maintain good > phasing between channels. The analog devices chip is <$50 so a custom > solution should be <$500/reference but with considerable development time. > >> > >> Bob Stricklin > >> > >> > >>> On May 4, 2015, at 10:02 PM, Jack Hickish > wrote: > >>> > >>> Hi John, > >>> > >>> Thanks for the info. I'll add Litelink to my list of suppliers to > investigate. > >>> We have no particular urge to multiplex the signals on to the fiber > unless there's a particularly neat/cheap solution to do that. There's no > great appetite to go custom. We've got about ~30 nodes, and my first stab > at getting an off-the-shelf solution turned up at a few k$ / node, not > including any cleanup electronics. > >>> > >>> Thanks again, > >>> > >>> Jack > >>> > >>> On Mon, 4 May 2015 at 19:25 John Ford wrote: > Hi CASPERites, > > For HERA, we're looking at distributing timing signals (PPS & 10Mhz > ref or > 500 MHz clock) over O(100m) fibers to various digitization nodes. > I figure some folks in CASPERland have experience with this kind of > system? > Did you use custom RF-over-fiber kit, or off-the-shelf PPS/10MHz > solutions? > Any words of wisdom/caution to share? > > Any responses much appreciated! > >>> > >>> We have several different schemes for the different signals. Are you > >>> planning for one fiber per signal per node? or one fiber with the > signals > >>> multiplexed on them? > >>> > >>> If the signals are one per signal, you can use some off-the-shelf > >>> solutions, but they are kind of pricey, and if you have a lot of nodes > to > >>> supply, it might be worth working on something custom. We have used > Math > >>> Associates stuff for this kind of work. Math Associates is now > litelink, > >>> and they tout the affordability of their stuff, so maybe it's > >>> reasonable... > >>> > >>> > >>> On the 10 MHz, we send the 10 MHz reference over fiber, and at the far > end > >>> use a crystal oscillator locked to the reference to clean up the noise > >>> from the fiber electronics. This is essential for interferometry, but > >>> maybe not for single-dish use. > >>> > >>> John > >>> > > Jack > > >>> > >>> > >> > > > > > > >
Re: [casper] Timing distribution over fiber
White rabbit is the version of IEEE1588 that locks bit clocks. Normal IEEE1588 doesn't do this. For this reason, (last I looked), there was no 10G implementation. 1G speeds is the highest White Rabbit implementation. The HW support for "normal" IEEE1588 is used to timestamp the packets without software in the loop (where the CPU will introduce jitter). But you don't necessarily need any special HW for IEEE1588... it can all be done in a normal FPGA using the standard Xilinx 10G IP core. There are FPGA cores available for the IEEE1588 part, so you don't have to implement it yourself. We considered this for MeerKAT at one stage, but in the end we couldn't achieve the required performance. It might be quite workable for HERA, though. Anyway, just a thought. It'd save you buying special HW and running additional fibres, if it meets your performance targets. Jason Manley CBF Manager SKA-SA Cell: +27 82 662 7726 Work: +27 21 506 7300 On 05 May 2015, at 18:49, Dan Werthimer wrote: > > > hi dave, > > i also think distributing clock and 1 PPS is simpler than IEEE1588. > > some of the IEEE1588 and white rabbit experts are here at berkeley. > see for example: > http://chess.eecs.berkeley.edu/pubs/881/dreams.pdf > > my limited understanding is that 1588 phase locks the bit clocks of > routers and switches together. 1588 routers and switches have SMA > connectors on them so you can use external maser/rubidium/GPS references. > > you can achieve spectacular accuracy and stabilitity with white rabbit > if you employ really good oscillators at each node, > i think white rabbit can acheive 70 picosecond RMS time transfer. > > best, > > dan > > > > > > On Tue, May 5, 2015 at 8:52 AM, David MacMahon > wrote: > Hi, Jason, > > I have a great deal of curiosity about IEEE-1588, but I've always wondered > about the precision/stability that's attainable. Compared with multiple > sample clocks, correlating signals sampled with one common clock seems far > more forgiving vis a vis clock frequency/phase stability. If you or John > could point me to any information about this, please do! > > Thanks, > Dave > > On May 4, 2015, at 11:44 PM, Jason Manley wrote: > > > On the far end of the concept spectrum, have you considered distributing > > time over your existing ethernet network with IEEE-1588, and using this to > > discipline local ovenised 10MHz oscs at each antenna? > > > > I'm cc'ing Johan Burger, who heads up our Timing and Frequency Reference > > subsystem, who might be able to offer some additional insight. I know > > they've tried a few different lasers and detectors, with varying levels of > > success. > > > > Jason Manley > > CBF Manager > > SKA-SA > > > > Cell: +27 82 662 7726 > > Work: +27 21 506 7300 > > > > On 05 May 2015, at 5:18, Bob Stricklin wrote: > > > >> Hi Jack and John, > >> > >> I wanted to add an input here….. > >> > >> I am working on a 10 MHz GPS slaved reference for my personal use. I am > >> working with a Analog Devices AD9548 Evaluation board (~$250) , GPS with 1 > >> PPS, and a ovenized 10 MHz osc. I also plan to distribute this clock and > >> have considered the Avago fiber product line. One of the older generation > >> Avago fiber parts should work fine for <$25 per channel. With careful > >> control of lengths and delays it should be possible to maintain good > >> phasing between channels. The analog devices chip is <$50 so a custom > >> solution should be <$500/reference but with considerable development time. > >> > >> Bob Stricklin > >> > >> > >>> On May 4, 2015, at 10:02 PM, Jack Hickish wrote: > >>> > >>> Hi John, > >>> > >>> Thanks for the info. I'll add Litelink to my list of suppliers to > >>> investigate. > >>> We have no particular urge to multiplex the signals on to the fiber > >>> unless there's a particularly neat/cheap solution to do that. There's no > >>> great appetite to go custom. We've got about ~30 nodes, and my first stab > >>> at getting an off-the-shelf solution turned up at a few k$ / node, not > >>> including any cleanup electronics. > >>> > >>> Thanks again, > >>> > >>> Jack > >>> > >>> On Mon, 4 May 2015 at 19:25 John Ford wrote: > Hi CASPERites, > > For HERA, we're looking at distributing timing signals (PPS & 10Mhz ref > or > 500 MHz clock) over O(100m) fibers to various digitization nodes. > I figure some folks in CASPERland have experience with this kind of > system? > Did you use custom RF-over-fiber kit, or off-the-shelf PPS/10MHz > solutions? > Any words of wisdom/caution to share? > > Any responses much appreciated! > >>> > >>> We have several different schemes for the different signals. Are you > >>> planning for one fiber per signal per node? or one fiber with the signals > >>> multiplexed on them? > >>> > >>> If the signals are one per signal, you can use some off-the-shelf > >>> solutions, but they are kind of pricey, and if you ha
Re: [casper] Timing distribution over fiber
Hi We have a complete White Rabbit setup, including 5km fibre test reels. It is driven by a Meinberg Plenum 1 clock. We are quite happy to chat with anyone on this topic of phase coherence. As you all know, the hardware and firmware is all Open Source, so as Jason mentions, easy to incorporate. I think I saw that the new CASPER card has provision for WR? Members of the team have specialised GPSDO systems developed in the lab that seem to give below 10 ns of jitter. Someone has tested it over 900km and beyond in Norway and sub ps jitter is achieved. One drawback is it needs a 1Ge fibre link, which I guess means, these days, its own fibre. We spoke to CERN about developing 10Ge and above, but it looks complex due to the many channels embedded in these streams. There has been some good work on sending it over microwave, but I guess that is a no-no for astronomy reserves. Regards On 5 May 2015 at 19:27, Jason Manley wrote: > White rabbit is the version of IEEE1588 that locks bit clocks. Normal > IEEE1588 doesn't do this. For this reason, (last I looked), there was no > 10G implementation. 1G speeds is the highest White Rabbit implementation. > The HW support for "normal" IEEE1588 is used to timestamp the packets > without software in the loop (where the CPU will introduce jitter). > > But you don't necessarily need any special HW for IEEE1588... it can all > be done in a normal FPGA using the standard Xilinx 10G IP core. There are > FPGA cores available for the IEEE1588 part, so you don't have to implement > it yourself. We considered this for MeerKAT at one stage, but in the end we > couldn't achieve the required performance. It might be quite workable for > HERA, though. Anyway, just a thought. It'd save you buying special HW and > running additional fibres, if it meets your performance targets. > > Jason Manley > CBF Manager > SKA-SA > > Cell: +27 82 662 7726 > Work: +27 21 506 7300 > > On 05 May 2015, at 18:49, Dan Werthimer wrote: > > > > > > > hi dave, > > > > i also think distributing clock and 1 PPS is simpler than IEEE1588. > > > > some of the IEEE1588 and white rabbit experts are here at berkeley. > > see for example: > > http://chess.eecs.berkeley.edu/pubs/881/dreams.pdf > > > > my limited understanding is that 1588 phase locks the bit clocks of > > routers and switches together. 1588 routers and switches have SMA > > connectors on them so you can use external maser/rubidium/GPS references. > > > > you can achieve spectacular accuracy and stabilitity with white rabbit > > if you employ really good oscillators at each node, > > i think white rabbit can acheive 70 picosecond RMS time transfer. > > > > best, > > > > dan > > > > > > > > > > > > On Tue, May 5, 2015 at 8:52 AM, David MacMahon < > dav...@astro.berkeley.edu> wrote: > > Hi, Jason, > > > > I have a great deal of curiosity about IEEE-1588, but I've always > wondered about the precision/stability that's attainable. Compared with > multiple sample clocks, correlating signals sampled with one common clock > seems far more forgiving vis a vis clock frequency/phase stability. If you > or John could point me to any information about this, please do! > > > > Thanks, > > Dave > > > > On May 4, 2015, at 11:44 PM, Jason Manley wrote: > > > > > On the far end of the concept spectrum, have you considered > distributing time over your existing ethernet network with IEEE-1588, and > using this to discipline local ovenised 10MHz oscs at each antenna? > > > > > > I'm cc'ing Johan Burger, who heads up our Timing and Frequency > Reference subsystem, who might be able to offer some additional insight. I > know they've tried a few different lasers and detectors, with varying > levels of success. > > > > > > Jason Manley > > > CBF Manager > > > SKA-SA > > > > > > Cell: +27 82 662 7726 > > > Work: +27 21 506 7300 > > > > > > On 05 May 2015, at 5:18, Bob Stricklin wrote: > > > > > >> Hi Jack and John, > > >> > > >> I wanted to add an input here….. > > >> > > >> I am working on a 10 MHz GPS slaved reference for my personal use. I > am working with a Analog Devices AD9548 Evaluation board (~$250) , GPS with > 1 PPS, and a ovenized 10 MHz osc. I also plan to distribute this clock and > have considered the Avago fiber product line. One of the older generation > Avago fiber parts should work fine for <$25 per channel. With careful > control of lengths and delays it should be possible to maintain good > phasing between channels. The analog devices chip is <$50 so a custom > solution should be <$500/reference but with considerable development time. > > >> > > >> Bob Stricklin > > >> > > >> > > >>> On May 4, 2015, at 10:02 PM, Jack Hickish > wrote: > > >>> > > >>> Hi John, > > >>> > > >>> Thanks for the info. I'll add Litelink to my list of suppliers to > investigate. > > >>> We have no particular urge to multiplex the signals on to the fiber > unless there's a particularly neat/cheap solution to do that. There's no > great appetite to go custom. We've got a
Re: [casper] Timing distribution over fiber
On Wed, 6 May 2015 at 07:35 Johan Burger wrote: > > Hi there, > > Do you maybe have any idea of requirement specifications for the HERA's RF > phase stability and time (?) - this might determine what technology could > be appropriate. > Hi Johan, Thanks for your response. We're sampling at 500 MHz, so we'd like to have a stability of "few" degrees, preferably over timescales of many hours but perhaps more reasonably on a calibration cadence of O(10 minutes) PPS is not such a big deal, and synchronization to a couple of ADC clock cycles is probably fine. We're investigating simple-ish ways to calibrate these out with signal injection. > > We at SKA Africa have after some iteration come up, with a precision RF > distribution system for many antennas. The type of laser and integrated > modulator have been proven in the field on large arrays (not just > MeerKAT). The RF can be directly transmitted (in our case up to 2-3 GHz > limited by our synthesizer - the precise frequency is 1.712GHz). 500MHz RF > over fibre can be done by this as well. There is conditioning of the RF > taking place on MeerKAT at the receiving end. As Jason said, not any or all > modules really do the job properly - we converged on a solution after > testing, that implicitly included modules evaluated from KAT-7 days, and > more recent modules from other manufacturers. > > "Low precision" timing ~100ns can indeed be done using PTP. If PPS is > required instead of an Ethernet package a special conversion board (PCIe) > is necessary. This is really enough for fringe finding - used in MeerKAT > S-band for example. That digitiser is mounted in an RFI shielded pedestal > of the antenna though. We supply the high precision PPS using our custom > system as described below. > > For our L-band digitisers mounted on the outside we had to come up with > special low power, low cost, high accuracy solution - this is being > implemented by Renier and Etienne and others here at SKA Africa (so a joint > effort by our time and frequency and digitiser team). The reason is that > White Rabbit is not compatible with 10Gbe links used on this system. > Furthermore Ethernet is actually quite noisy as per MeerKAT measurements, > and White Rabbit and PTP uses that (and with highish power consumption and > largish board size), and is not preferable in a high purity clock signal > and PPS module. We found that measurement based PPS system will meet our > requirements though, for stabilized links and provides us with accurate > absolute time references at antennas, using analog methodologies. This for > example being important in pulsar science. > > I am not sure what level of RFI shielding you would be able to mount > around modules, but as said RFI from Ethernet has certainly been found to > be an RFI culprit, and cannot be therefore be used in MeerKAT close to > sensitive modules - and needs to separately shielded. This therefore means > that if PPS is generated from White Rabbit/PTP there is still some > uncertain propagation paths left (important at least for MeerKAT) up to the > point of digitization where a timing edge is inserted. We are using > seperate fibres for PPS and RF, to further limit self-RFI and as it was > found that requirements could only be met in this way. > This is a good point, and something we'll make sure to keep in mind... Thanks again, Jack > > Regards > > Johan Burger > > > > > > For MeerKAT high precision timing a special PPS solution is used. There > are seperate PPS transmitters and > On Tue, May 5, 2015 at 7:56 PM, Michael Inggs wrote: > >> Hi >> >> We have a complete White Rabbit setup, including 5km fibre test reels. It >> is driven by a Meinberg Plenum 1 clock. We are quite happy to chat with >> anyone on this topic of phase coherence. As you all know, the hardware and >> firmware is all Open Source, so as Jason mentions, easy to incorporate. I >> think I saw that the new CASPER card has provision for WR? Members of the >> team have specialised GPSDO systems developed in the lab that seem to give >> below 10 ns of jitter. >> >> Someone has tested it over 900km and beyond in Norway and sub ps jitter >> is achieved. >> >> One drawback is it needs a 1Ge fibre link, which I guess means, these >> days, its own fibre. We spoke to CERN about developing 10Ge and above, but >> it looks complex due to the many channels embedded in these streams. There >> has been some good work on sending it over microwave, but I guess that is a >> no-no for astronomy reserves. >> >> Regards >> >> >> >> On 5 May 2015 at 19:27, Jason Manley wrote: >> >>> White rabbit is the version of IEEE1588 that locks bit clocks. Normal >>> IEEE1588 doesn't do this. For this reason, (last I looked), there was no >>> 10G implementation. 1G speeds is the highest White Rabbit implementation. >>> The HW support for "normal" IEEE1588 is used to timestamp the packets >>> without software in the loop (where the CPU will introduce jitter). >>> >>> But
Re: [casper] Timing distribution over fiber
Jason, Mike, Jonathan, others!, White rabbit, and PTP are both pretty interesting. My suspicion is that it may end up being no more expensive and less worrysome (if not necessarily actually better) to distribute PPS/ some ref. clock directly, but HERA is a good excuse to investigate these alternative solutions. Mike -- there's much reading I need to do about White Rabbit before I can usefully have a conversation with anyone about it, but at some point in the future I'd certainly be interested in taking you up on the offer to chat more about your system. Thanks again Jack On Tue, 5 May 2015 at 10:57 Michael Inggs wrote: > Hi > > We have a complete White Rabbit setup, including 5km fibre test reels. It > is driven by a Meinberg Plenum 1 clock. We are quite happy to chat with > anyone on this topic of phase coherence. As you all know, the hardware and > firmware is all Open Source, so as Jason mentions, easy to incorporate. I > think I saw that the new CASPER card has provision for WR? Members of the > team have specialised GPSDO systems developed in the lab that seem to give > below 10 ns of jitter. > > Someone has tested it over 900km and beyond in Norway and sub ps jitter is > achieved. > > One drawback is it needs a 1Ge fibre link, which I guess means, these > days, its own fibre. We spoke to CERN about developing 10Ge and above, but > it looks complex due to the many channels embedded in these streams. There > has been some good work on sending it over microwave, but I guess that is a > no-no for astronomy reserves. > > Regards > > > > On 5 May 2015 at 19:27, Jason Manley wrote: > >> White rabbit is the version of IEEE1588 that locks bit clocks. Normal >> IEEE1588 doesn't do this. For this reason, (last I looked), there was no >> 10G implementation. 1G speeds is the highest White Rabbit implementation. >> The HW support for "normal" IEEE1588 is used to timestamp the packets >> without software in the loop (where the CPU will introduce jitter). >> >> But you don't necessarily need any special HW for IEEE1588... it can all >> be done in a normal FPGA using the standard Xilinx 10G IP core. There are >> FPGA cores available for the IEEE1588 part, so you don't have to implement >> it yourself. We considered this for MeerKAT at one stage, but in the end we >> couldn't achieve the required performance. It might be quite workable for >> HERA, though. Anyway, just a thought. It'd save you buying special HW and >> running additional fibres, if it meets your performance targets. >> >> Jason Manley >> CBF Manager >> SKA-SA >> >> Cell: +27 82 662 7726 >> Work: +27 21 506 7300 >> >> On 05 May 2015, at 18:49, Dan Werthimer wrote: >> >> > >> > >> > hi dave, >> > >> > i also think distributing clock and 1 PPS is simpler than IEEE1588. >> > >> > some of the IEEE1588 and white rabbit experts are here at berkeley. >> > see for example: >> > http://chess.eecs.berkeley.edu/pubs/881/dreams.pdf >> > >> > my limited understanding is that 1588 phase locks the bit clocks of >> > routers and switches together. 1588 routers and switches have SMA >> > connectors on them so you can use external maser/rubidium/GPS >> references. >> > >> > you can achieve spectacular accuracy and stabilitity with white rabbit >> > if you employ really good oscillators at each node, >> > i think white rabbit can acheive 70 picosecond RMS time transfer. >> > >> > best, >> > >> > dan >> > >> > >> > >> > >> > >> > On Tue, May 5, 2015 at 8:52 AM, David MacMahon < >> dav...@astro.berkeley.edu> wrote: >> > Hi, Jason, >> > >> > I have a great deal of curiosity about IEEE-1588, but I've always >> wondered about the precision/stability that's attainable. Compared with >> multiple sample clocks, correlating signals sampled with one common clock >> seems far more forgiving vis a vis clock frequency/phase stability. If you >> or John could point me to any information about this, please do! >> > >> > Thanks, >> > Dave >> > >> > On May 4, 2015, at 11:44 PM, Jason Manley wrote: >> > >> > > On the far end of the concept spectrum, have you considered >> distributing time over your existing ethernet network with IEEE-1588, and >> using this to discipline local ovenised 10MHz oscs at each antenna? >> > > >> > > I'm cc'ing Johan Burger, who heads up our Timing and Frequency >> Reference subsystem, who might be able to offer some additional insight. I >> know they've tried a few different lasers and detectors, with varying >> levels of success. >> > > >> > > Jason Manley >> > > CBF Manager >> > > SKA-SA >> > > >> > > Cell: +27 82 662 7726 >> > > Work: +27 21 506 7300 >> > > >> > > On 05 May 2015, at 5:18, Bob Stricklin wrote: >> > > >> > >> Hi Jack and John, >> > >> >> > >> I wanted to add an input here….. >> > >> >> > >> I am working on a 10 MHz GPS slaved reference for my personal use. I >> am working with a Analog Devices AD9548 Evaluation board (~$250) , GPS with >> 1 PPS, and a ovenized 10 MHz osc. I also plan to distribute this clock and >> have considere
Re: [casper] Timing distribution over fiber
Hi Bob, Sounds interesting -- for what it's worth, I've just learnt some fellow astronomers in Italy (Andrea Maccaferri at INAF's Medicina observatory) have built some PPS/refclk distribution systems based on the Avago fiber parts -- perhaps you might be interested in getting in touch with him... Thanks (to everyone on this thread) Jack On Mon, 4 May 2015 at 20:19 Bob Stricklin wrote: > Hi Jack and John, > > I wanted to add an input here….. > > I am working on a 10 MHz GPS slaved reference for my personal use. I am > working with a Analog Devices AD9548 Evaluation board (~$250) , GPS with 1 > PPS, and a ovenized 10 MHz osc. I also plan to distribute this clock and > have considered the Avago fiber product line. One of the older generation > Avago fiber parts should work fine for <$25 per channel. With careful > control of lengths and delays it should be possible to maintain good > phasing between channels. The analog devices chip is <$50 so a custom > solution should be <$500/reference but with considerable development time. > > Bob Stricklin > > > On May 4, 2015, at 10:02 PM, Jack Hickish wrote: > > Hi John, > > Thanks for the info. I'll add Litelink to my list of suppliers to > investigate. > We have no particular urge to multiplex the signals on to the fiber unless > there's a particularly neat/cheap solution to do that. There's no great > appetite to go custom. We've got about ~30 nodes, and my first stab at > getting an off-the-shelf solution turned up at a few k$ / node, not > including any cleanup electronics. > > Thanks again, > > Jack > > On Mon, 4 May 2015 at 19:25 John Ford wrote: > >> > Hi CASPERites, >> > >> > For HERA, we're looking at distributing timing signals (PPS & 10Mhz ref >> or >> > 500 MHz clock) over O(100m) fibers to various digitization nodes. >> > I figure some folks in CASPERland have experience with this kind of >> > system? >> > Did you use custom RF-over-fiber kit, or off-the-shelf PPS/10MHz >> > solutions? >> > Any words of wisdom/caution to share? >> > >> > Any responses much appreciated! >> >> We have several different schemes for the different signals. Are you >> planning for one fiber per signal per node? or one fiber with the signals >> multiplexed on them? >> >> If the signals are one per signal, you can use some off-the-shelf >> solutions, but they are kind of pricey, and if you have a lot of nodes to >> supply, it might be worth working on something custom. We have used Math >> Associates stuff for this kind of work. Math Associates is now litelink, >> and they tout the affordability of their stuff, so maybe it's >> reasonable... >> >> >> On the 10 MHz, we send the 10 MHz reference over fiber, and at the far end >> use a crystal oscillator locked to the reference to clean up the noise >> from the fiber electronics. This is essential for interferometry, but >> maybe not for single-dish use. >> >> John >> >> > >> > Jack >> > >> >> >> >
