Re: [etherlab-dev] EoE patchs and questions

[Esben Haabendal]

Graeme Foot  writes:

> I agree that EtherCAT is dynamic in that it allows Hot Plugging of
> modules.  But every time a module is added or removed the system is
> rescanned, all slaves are put into PREOP, Distributed clocks delays
> are recalculated etc then all configured slaves put back into OP.
>
> So although you can dynamically plug in modules at any time, your
> application first of all needs to configure expected modules, and if
> they exist know what to do with them.  So at some point your
> application needs to know all possible modules that could be plugged
> in.  (EoE modules are slightly different though in that they also work
> in PREOP.)
>
> You say you don't use aliases on your EoE modules, but do you use
> aliases on the rest?  (or at least the couplers?)  If not how do you
> know what is currently plugged in?

I think most projects here currently do not use alias.  They probably
should do that for our couplers.  But that does not change that I
believe that etherlabmaster should fully support systems that does not
use alias.

/Esben
___
etherlab-dev mailing list
etherlab-dev@etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-dev

Re: [etherlab-dev] EoE patchs and questions

[Esben Haabendal]

"Gavin Lambert"  writes:

> Those sound like great changes to have.

Hi Gavin

Please be a aware, that 0002-eoe-via-rtdm.patch in this patch series
reverts base/0018-Use-call-back-functions.patch in your current
patchset.

This will break existing non-RTDM user-space applications using EoE.
And the solution provided instead does only apply to RTDM applications.

/Esben
___
etherlab-dev mailing list
etherlab-dev@etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-dev

Re: [etherlab-dev] EoE patchs and questions

[Graeme Foot]

Hi,

I agree that EtherCAT is dynamic in that it allows Hot Plugging of modules.  
But every time a module is added or removed the system is rescanned, all slaves 
are put into PREOP, Distributed clocks delays are recalculated etc then all 
configured slaves put back into OP.

So although you can dynamically plug in modules at any time, your application 
first of all needs to configure expected modules, and if they exist know what 
to do with them.  So at some point your application needs to know all possible 
modules that could be plugged in.  (EoE modules are slightly different though 
in that they also work in PREOP.)

You say you don't use aliases on your EoE modules, but do you use aliases on 
the rest?  (or at least the couplers?)   If not how do you know what is 
currently plugged in?

Regards,
Graeme.

-Original Message-
From: Esben Haabendal [mailto:esbenhaaben...@gmail.com] On Behalf Of Esben 
Haabendal
Sent: Friday, 2 March 2018 10:50 PM
To: Graeme Foot 
Cc: etherlab-dev@etherlab.org
Subject: Re: [etherlab-dev] EoE patchs and questions

Graeme Foot  writes:

>> -Original Message-
>> From: Esben Haabendal [mailto:esbenhaaben...@gmail.com] On Behalf Of 
>> Esben Haabendal
>> Sent: Thursday, 1 March 2018 2:02 AM
>> To: Graeme Foot 
>> Cc: etherlab-dev@etherlab.org
>> Subject: Re: [etherlab-dev] EoE patchs and questions
>> 
>> Hi Graeme
>> 
>> Graeme Foot  writes:
>> > So the first patch does a few things:
>> >
>> > 1) adds explicit eoe_addif and eoe_delif tool functions so that you
>> >can manually add/remove an eoe iface without the slave existing
>> > 2) no longer deletes and eoe iface if the slave disappears
>> 
>> This is a behavioral change compared to how eoe interface work now.  
>> And it will break our current system, which relies on the interfaces 
>> being created and deleted together with the slaves.
>
> The changes have added a couple of settings to the ethercat config 
> file (EOE_INTERFACES and EOE_AUTOCREATE).  The default for 
> EOE_AUTOCREATE is true, so eoe interfaces will be autocreated as EOE 
> slaves are connected.  The only "breaking" behavioral change here is 
> that they will no longer be deleted when the slave is removed.

Yes, I see that.

> The only issue I have with this is that if you connect a slave that 
> does not yet have an alias set, it will auto-create using position 
> naming (e.g. eoe0s4).

We don't use aliases, but I have other issues with the change.

We have created a daemon which listens to netlink(7) messages, taking 
appropriate actions when interfaces come and go, links go up and down.
This is used to setup all network related configurations, including adding eoe 
interfaces to bridges.

This will obviously not continue to work as it does if eoe interfaces suddenly 
is not deleted when slaves disappear.

> Once the alias is set it will auto-create using the alias naming (e.g. 
> eoe0a16000) but keep the position named eoe port.  As this is usually 
> at setup and configuration time then you are likely to reboot and it 
> will be gone.  (Or you can explicitly call the eoe_delif
> function.)
>
>> 
>> With EtherCAT being a dynamic structure, I don't see this new way is 
>> supposed to work in general use-case, where the bus configuration is 
>> not guaranteed to be fixed.
>
> When the EOE slave is removed the port transitions to a disconnected 
> state (like pulling the plug out of a normal network card).  My 
> (unfounded) feeling is that this would be preferable to other software 
> using the port rather than deleting the interface all together.

This way of looking at it implies a static EtherCAT bus configuration.
By nature, EtherCAT is dynamic.  Whenever changes are detected, the bus is 
rescanned, and slaves are added / deleted accordingly.

When the bus configuration is changed, the new configuration might not be 
wrong, so insisting on the previous configuration could therefore be 
problematic.

> It also means that if you have added the port to a bridge (my use
> case) then it will remain in the bridge and resume once it is plugged 
> back in again.
> If it is deleted then it is removed from the bridge and the next time 
> it is plugged in again it will need to be manually re-added it to the 
> bridge.

As described above, we handle this in the dynamic case.  If an interface is 
deleted, it is naturally removed from the bridge.  If it is added again, it 
will added back to the bridge.

> I would think that even in a dynamic structure if an EOE module has 
> been plugged in once it is likely to be plugged in again at some 
> stage.

Maybe, maybe not.  It really depends on the use-case.

>> Would it be possible to make this new behavior optional, so that t

Re: [etherlab-dev] EoE patchs and questions

[Esben Haabendal]

Graeme Foot  writes:

>> -Original Message-
>> From: Esben Haabendal [mailto:esbenhaaben...@gmail.com] On Behalf Of Esben 
>> Haabendal
>> Sent: Thursday, 1 March 2018 2:02 AM
>> To: Graeme Foot 
>> Cc: etherlab-dev@etherlab.org
>> Subject: Re: [etherlab-dev] EoE patchs and questions
>> 
>> Hi Graeme
>> 
>> Graeme Foot  writes:
>> > So the first patch does a few things:
>> >
>> > 1) adds explicit eoe_addif and eoe_delif tool functions so that you
>> >can manually add/remove an eoe iface without the slave existing
>> > 2) no longer deletes and eoe iface if the slave disappears
>> 
>> This is a behavioral change compared to how eoe interface work now.  And it
>> will break our current system, which relies on the interfaces being created
>> and deleted together with the slaves.
>
> The changes have added a couple of settings to the ethercat config file
> (EOE_INTERFACES and EOE_AUTOCREATE).  The default for EOE_AUTOCREATE is true,
> so eoe interfaces will be autocreated as EOE slaves are connected.  The only
> "breaking" behavioral change here is that they will no longer be deleted when
> the slave is removed.

Yes, I see that.

> The only issue I have with this is that if you connect a slave that
> does not yet have an alias set, it will auto-create using position
> naming (e.g. eoe0s4).

We don't use aliases, but I have other issues with the change.

We have created a daemon which listens to netlink(7) messages, taking
appropriate actions when interfaces come and go, links go up and down.
This is used to setup all network related configurations, including
adding eoe interfaces to bridges.

This will obviously not continue to work as it does if eoe interfaces
suddenly is not deleted when slaves disappear.

> Once the alias is set it will auto-create using the alias naming
> (e.g. eoe0a16000) but keep the position named eoe port.  As this is
> usually at setup and configuration time then you are likely to reboot
> and it will be gone.  (Or you can explicitly call the eoe_delif
> function.)
>
>> 
>> With EtherCAT being a dynamic structure, I don't see this new way is
>> supposed to work in general use-case, where the bus configuration is
>> not guaranteed to be fixed.
>
> When the EOE slave is removed the port transitions to a disconnected state
> (like pulling the plug out of a normal network card).  My (unfounded) feeling
> is that this would be preferable to other software using the port rather than
> deleting the interface all together.

This way of looking at it implies a static EtherCAT bus configuration.
By nature, EtherCAT is dynamic.  Whenever changes are detected, the bus
is rescanned, and slaves are added / deleted accordingly.

When the bus configuration is changed, the new configuration might not
be wrong, so insisting on the previous configuration could therefore be
problematic.

> It also means that if you have added the port to a bridge (my use
> case) then it will remain in the bridge and resume once it is plugged
> back in again.
> If it is deleted then it is removed from the bridge and the next time
> it is plugged in again it will need to be manually re-added it to the
> bridge.

As described above, we handle this in the dynamic case.  If an interface
is deleted, it is naturally removed from the bridge.  If it is added
again, it will added back to the bridge.

> I would think that even in a dynamic structure if an EOE module has
> been plugged in once it is likely to be plugged in again at some
> stage.

Maybe, maybe not.  It really depends on the use-case.

>> Would it be possible to make this new behavior optional, so that the
>> default behavior is as it is now?
>
> Sure.  Two options:
> 1) Have another ethercat config file option: EOE_AUTODELETE (default true)
> 2) Have EOE_AUTOCREATE also mean that it will auto-delete.
>
> I prefer option 2 for ease of implementation, though option 1 would be better
> if my assumptions above are correct.  Try the attached completely untested
> (and manually edited) patch file for option 2.

I think I prefer option 2.  I think it could cause to much trouble to
enable auto-create without auto-delete.

Think for example of a system where slaves have separate power supplies.
When powering up the system, you might have some temporary bus
configurations until all slaves are powered up.  If you have auto-create
and not auto-delete, you will risk having random garbage eoe interfaces
created during that.

>
>> 
>> /Esben
>>
>
> I'm interested to know what your application is to have such a dynamic 
> structure.

Let me answer that in two ways.

1. I really believe EtherCAT is a protocol with an inherent dynamic bus

Re: [etherlab-dev] EoE patchs and questions

[Graeme Foot]

> -Original Message-
> From: Esben Haabendal [mailto:esbenhaaben...@gmail.com] On Behalf Of Esben 
> Haabendal
> Sent: Thursday, 1 March 2018 2:02 AM
> To: Graeme Foot 
> Cc: etherlab-dev@etherlab.org
> Subject: Re: [etherlab-dev] EoE patchs and questions
> 
> Hi Graeme
> 
> Graeme Foot  writes:
> > So the first patch does a few things:
> >
> > 1) adds explicit eoe_addif and eoe_delif tool functions so that you
> >can manually add/remove an eoe iface without the slave existing
> > 2) no longer deletes and eoe iface if the slave disappears
> 
> This is a behavioral change compared to how eoe interface work now.  And it 
> will break our current system, which relies on the interfaces being created 
> and deleted together with the slaves.

The changes have added a couple of settings to the ethercat config file 
(EOE_INTERFACES and EOE_AUTOCREATE).  The default for EOE_AUTOCREATE is true, 
so eoe interfaces will be autocreated as EOE slaves are connected.  The only 
"breaking" behavioral change here is that they will no longer be deleted when 
the slave is removed.  The only issue I have with this is that if you connect a 
slave that does not yet have an alias set, it will auto-create using position 
naming (e.g. eoe0s4).  Once the alias is set it will auto-create using the 
alias naming (e.g. eoe0a16000) but keep the position named eoe port.  As this 
is usually at setup and configuration time then you are likely to reboot and it 
will be gone.  (Or you can explicitly call the eoe_delif function.)

> 
> With EtherCAT being a dynamic structure, I don't see this new way is supposed 
> to work in general use-case, where the bus configuration is not guaranteed to 
> be fixed.

When the EOE slave is removed the port transitions to a disconnected state 
(like pulling the plug out of a normal network card).  My (unfounded) feeling 
is that this would be preferable to other software using the port rather than 
deleting the interface all together.  It also means that if you have added the 
port to a bridge (my use case) then it will remain in the bridge and resume 
once it is plugged back in again.  If it is deleted then it is removed from the 
bridge and the next time it is plugged in again it will need to be manually 
re-added it to the bridge.

I would think that even in a dynamic structure if an EOE module has been 
plugged in once it is likely to be plugged in again at some stage.

> 
> Would it be possible to make this new behavior optional, so that the default 
> behavior is as it is now?

Sure.  Two options:
1) Have another ethercat config file option: EOE_AUTODELETE (default true)
2) Have EOE_AUTOCREATE also mean that it will auto-delete.

I prefer option 2 for ease of implementation, though option 1 would be better 
if my assumptions above are correct.  Try the attached completely untested (and 
manually edited) patch file for option 2.

> 
> /Esben
>

I'm interested to know what your application is to have such a dynamic 
structure.


Regards,
Graeme.


0001b-eoe-addif-delif-tools.patch
Description: 0001b-eoe-addif-delif-tools.patch
___
etherlab-dev mailing list
etherlab-dev@etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-dev

Re: [etherlab-dev] EoE patchs and questions

[Esben Haabendal]

Hi Graeme

Graeme Foot  writes:
> So the first patch does a few things:
>
> 1) adds explicit eoe_addif and eoe_delif tool functions so that you
>can manually add/remove an eoe iface without the slave existing
> 2) no longer deletes and eoe iface if the slave disappears

This is a behavioral change compared to how eoe interface work now.  And
it will break our current system, which relies on the interfaces being
created and deleted together with the slaves.

With EtherCAT being a dynamic structure, I don't see this new way is
supposed to work in general use-case, where the bus configuration is not
guaranteed to be fixed.

Would it be possible to make this new behavior optional, so that the
default behavior is as it is now?

/Esben
___
etherlab-dev mailing list
etherlab-dev@etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-dev

Re: [etherlab-dev] EoE patchs and questions

[Graeme Foot]

Hi,

I’m off from work for the weekend, but I did view the contents of 0x808 (size 
8) before and after doing a write of zero and all the values stayed the same. 
SM1 is configured with its default values so I’m assuming writing a zero will 
set it back to its defaults, so will remain the same. Needs more testing but 
ran out of time this week.

I got to the point of writing a zero to SM1 due to noticing that the rx 
communications resumed when I manually changed the state of the slave from OP 
to PREOP. So I followed the steps in the state machine. That was the command 
that did it. Other things I tried like rescans or unplugging the coupler 
(keeping power on the EoE slave didn’t work).

I don’t know of any slaves that might require OP mode to run as I’m new to EoE 
too. From my memory of the code on initial startup it remains in PREOP and its 
only on a master deactivate that it remains in OP (and it does so by putting 
EoE slaves back into OP after all slaves have been set to PREOP).

Time for more testing next week. Yes I will be  wiresharking the Ethercat 
frames. I’ve been checking the Ethernet side so far. Will need to grab another 
computer to check the Ethercat side.

Thanks,
Graeme.


From: Gavin Lambert 
Sent: Friday, 16 February 2018 7:47:17 PM
To: Graeme Foot; etherlab-dev@etherlab.org
Subject: RE: EoE patchs and questions

Those sound like great changes to have.

I suspect the EoE-OP thing came from an assumption that the slave had to be in 
OP to transfer EoE frames; there was previously a similar assumption regarding 
the DC reference clock that was fixed in 
[559f2f].
  I don’t have any experience with EoE myself but a quick glance through the 
manual for EL6614 does suggest that it will happily do EoE in PREOP and above.  
Do you think there could be any older slaves that might need OP for that?

The register write to 0x808 as a recovery from that condition seems a bit 
peculiar – most of those registers are read-only while SM1 is enabled – though 
you’re writing 0 to 0x80E, which should disable the SM, which then ought to 
stop it working entirely, unless something reconfigures it.

Perhaps inspecting other SM registers might be interesting?  Or see if there’s 
anything noticeable around that time in a Wireshark trace (if you have some way 
to detect exactly when it stops)?  Does the problem still happen with fewer 
patches applied?

From: Graeme Foot
Sent: Friday, 16 February 2018 19:01
To: etherlab-dev@etherlab.org
Subject: [etherlab-dev] EoE patchs and questions

Hi,

I've been setting up my system to use EoE (Ethernet over EtherCAT) with an RTAI 
user space application.

I've updated my master to revision 33b922ec1871 (default branch) and applied 
the gavinl (Gavin Lambert) patch set 20171108.
Linux 2.6.32.11
RTAI 3.8.1


Firstly I have a bit of a different use case for my EoE.  The current 
implementation auto creates and removes the eoe interfaces as the EoE capable 
slaves are configured and removed.  This means the interface is not available 
until the slave is scanned, and is not available if it is removed.  The eoe 
interface is also temporarily destroyed on a bus rescan.  In my use case I want 
to bridge the eoe interface to a real Ethernet interface.  So I want the eoe 
interface to always exists whether the slave is plugged in or not.

So the first patch does a few things:
1) adds explicit eoe_addif and eoe_delif tool functions so that you can 
manually add/remove an eoe iface without the slave existing
2) no longer deletes and eoe iface if the slave disappears
3) will relink a slave to an eoe iface when it is configured
4) will let you configure eoe ifaces via the sysconfig/ethercat config file
5) will let you turn off auto creation of eoe ifaces via the sysconfig/ethercat 
config file
6) no longer keeps slaves with EoE capability in OP mode when the master is 
deactivated

The above is made possible by using the netif_carrier_on() and 
netif_carrier_off() functions of the iface.  (The same as having a normal 
network interface up, but not plugged in.)

The other thing the patch does is fix a race condition bug in the eoe iface 
code.  The current implementation uses a struct list_head queue with a 
semaphore to protect it between the iface tx callback and the ethercat thread.  
Sleeps are not allowed in the ifaces tx callback as it is in an interrupt 
context.  To fix this I have changed the queue to a ring buffer so that it no 
longer needs a lock.

FYI, when the race condition occurred I was getting:
BUG: scheduling while atomic
Call Trace:
[] ? ktime_get_real+0x0/0x29
[] ? ktime_get+0x0/0x88

Florian you may be interested in this patch, especially the bug fix part.


The second patch is so that I can run the EoE pump without callbacks.  As I am 
using a user space RTAI application I cannot use callbacks as they would need 
to call back from a kern

Re: [etherlab-dev] EoE patchs and questions

[Gavin Lambert]

Those sound like great changes to have.

 

I suspect the EoE-OP thing came from an assumption that the slave had to be
in OP to transfer EoE frames; there was previously a similar assumption
regarding the DC reference clock that was fixed in
 [559f2f].  I don't have any experience with EoE myself but
a quick glance through the manual for EL6614 does suggest that it will
happily do EoE in PREOP and above.  Do you think there could be any older
slaves that might need OP for that?

 

The register write to 0x808 as a recovery from that condition seems a bit
peculiar - most of those registers are read-only while SM1 is enabled -
though you're writing 0 to 0x80E, which should disable the SM, which then
ought to stop it working entirely, unless something reconfigures it.

 

Perhaps inspecting other SM registers might be interesting?  Or see if
there's anything noticeable around that time in a Wireshark trace (if you
have some way to detect exactly when it stops)?  Does the problem still
happen with fewer patches applied?

 

From: Graeme Foot
Sent: Friday, 16 February 2018 19:01
To: etherlab-dev@etherlab.org
Subject: [etherlab-dev] EoE patchs and questions

 

Hi,

 

I've been setting up my system to use EoE (Ethernet over EtherCAT) with an
RTAI user space application.

 

I've updated my master to revision 33b922ec1871 (default branch) and applied
the gavinl (Gavin Lambert) patch set 20171108.

Linux 2.6.32.11

RTAI 3.8.1

 

 

Firstly I have a bit of a different use case for my EoE.  The current
implementation auto creates and removes the eoe interfaces as the EoE
capable slaves are configured and removed.  This means the interface is not
available until the slave is scanned, and is not available if it is removed.
The eoe interface is also temporarily destroyed on a bus rescan.  In my use
case I want to bridge the eoe interface to a real Ethernet interface.  So I
want the eoe interface to always exists whether the slave is plugged in or
not.

 

So the first patch does a few things:

1) adds explicit eoe_addif and eoe_delif tool functions so that you can
manually add/remove an eoe iface without the slave existing

2) no longer deletes and eoe iface if the slave disappears

3) will relink a slave to an eoe iface when it is configured

4) will let you configure eoe ifaces via the sysconfig/ethercat config file

5) will let you turn off auto creation of eoe ifaces via the
sysconfig/ethercat config file

6) no longer keeps slaves with EoE capability in OP mode when the master is
deactivated

 

The above is made possible by using the netif_carrier_on() and
netif_carrier_off() functions of the iface.  (The same as having a normal
network interface up, but not plugged in.)

 

The other thing the patch does is fix a race condition bug in the eoe iface
code.  The current implementation uses a struct list_head queue with a
semaphore to protect it between the iface tx callback and the ethercat
thread.  Sleeps are not allowed in the ifaces tx callback as it is in an
interrupt context.  To fix this I have changed the queue to a ring buffer so
that it no longer needs a lock.

 

FYI, when the race condition occurred I was getting:

BUG: scheduling while atomic

Call Trace:

[] ? ktime_get_real+0x0/0x29

[] ? ktime_get+0x0/0x88

 

Florian you may be interested in this patch, especially the bug fix part.

 

 

The second patch is so that I can run the EoE pump without callbacks.  As I
am using a user space RTAI application I cannot use callbacks as they would
need to call back from a kernel context to the user space context.  Instead
I am running a thread in my application that makes calls into EtherCAT in a
similar fashion to the masters EoE thread.  I have created two functions
(ecrt_master_eoe_is_open() and ecrt_master_eoe_process()) to call without
application locks as the locks only need to be around the
ecrt_master_receive() and ecrt_master_send_ext() calls.

 

 

Now for the question.  I have been hammering my test rig pretty hard with
various communications (pings with multiple fragments multiple times a
second from both directions, SDO calls to the EoE slave without a pause
approx. 100 per second).  Every now and then (after around 10 to 30 minutes
with the above tests) the receive mailbox (SM1) of the EoE slave stops
responding (slave to master).  CoE reads to the slave also fail.  The
transmit mailbox still continues to function.  The RX SM1 status register
continually returns a zero value.  I have found that if I send the command
below the receive mailbox starts to function again (until it doesn't):

 

  ethercat reg_write -p3 0x808 -tuint64 0

 

Has anyone else come across this?  At the moment I suspecting a Slave
firmware bug (EL6614).  Does anyone have any other ideas?

 

 

Regards,

Graeme Foot.

 

 

 

___
etherlab-dev mailing list
etherlab-dev@etherlab.org
ht