Re: [Kicad-developers] Ideas for a lightweight KiCad PDM

[Kevin Bortis]

On Tue, Dec 6, 2016 at 4:39 PM, Clemens Koller  wrote:
> Hello, Kevin!
>
> I fully support your ideas. In a commercial environment, a proper PDM
including version management and documentation is mandatory to be able to
deliver a product where you can realize a traceability down to the date
code of a component once your design is manufactured.
>
> But since all projects / infrastructures / companies differ, it is
propably not a good idea to hard code some workflow into the design tool
(KiCad) and expect a one fits all approach.
>
> I have sort of a wishlist which would allow me to interface KiCad to my
existing infrastucture and solve some issues which cost me a lot of time:
> - The library as well as all design data should be accessible
(read-writeable) like a database (ideally: while KiCad is running).
> - Everything (even every single connection in the schematics) has a
version and a (modification) timestamp and is sortable by object,
properties, version and timestamp.
> - A copy (reuse) of fractions of a design should optionally include
it's objects properties, constraints, versions and history.
> - A diff and merge of parts or whole designs would improve versioning and
design-reuse a lot.
> * I would love to see forward/rewind buttons for libraries, schematics
and layout similar to browsing through the git history of a software
project 8-).
>
> For performance reasons I believe it is not a good idea to implement
design to library connectivity with links on a file-system level as it is
done by bothy commercial products quite a lot. (One horribly expensive
product seems to hold the whole design in an (optimized) database in memory
and is able to replicate the data in real-time in between several
workstations to feature "concurrent design entry" where several
people/autorouters are able to work on one design at the same time.)
>
> OT:
> I am working a lot with databases (SQL based) and several external tools
(self-written stuff + git) to manage my component library, the library
generators, as well as my local component information / stock data. I am
able to "link" a BOM written by the design tool with my component database
(+ local stock data) and then with the i.e. SAP data (+ remote stock data)
of my manufacturer to transfer my design to the assembly in an automated
way. We implemented something like a closed-loop-verification process where
I am able to read back the SAP data (maybe in the future through some
remote database interface) into my database to verify that the data will
arrive correctly at the machines. Sometimes, I'll need to update my local
data when the assembly needs a change somewhere. Since everything is a
moving target you want to freeze all the information for each manufacturing
lot / product version.
>
> These processes different a lot from one assembly house to the next one.
So, there is a cookbook of scripts tailored to each manufacturers process
with a lot of verification steps built in. There are usually no more .xls
files envolved in the whole process where busy people tend to add typos
manually without getting noticed.
>
> Regards,
>
> Clemens
>

Thanks for the feedback. I will include your  these in my collection of
things to consider.

>
> On 2016-12-06 08:22, Kevin Bortis wrote:
>> Hi
>>
>> I try to figure out how it would be possible to create a central
>> storage for symbols, footprints and other design data. With central I
>> mean something like a design data server, where it is possible to
>> assign workflows with sign-offs, make proper versioning, assign order
>> numbers and define second sources. (PDM ultra light for KiCad).
>>
>> I currently work on some projects where we need to be sure that each
>> change on current products gets a proper review and documentation.
>> This always starts with a "Change Request", followed by a "Change
>> Order". To make sure everything is right, we currently use a very
>> expensive EDA suite with even more expensive PDM[2] software. Even if
>> this seems overkill for private projects, a little more lighter
>> version of this workflow makes sens for every company and even for
>> some bigger private projects. I helps to grantees some quality and
>> data integrity.
>>
>> After some analysis how KiCad does things I see some little problems.
>> The first of them is described in the following lines:
>>
>> Currently the footprint and soon[1] symbols will be assigned with a
>> "library nickname"/"symbol name" key. This is also true for the 3D
>> model inside the  footprint file. At the moment this is the only
>> reference to the library. If the component in the local file storage
>> changes, the symbol/footprint gets changed immediately in the project
>> and that is exactly what I want to avoid. To make a server based
>> library, each symbol or footprint would need a unique identifier and
>> some way to define the storage location. Say instead of define
>> somelibrary/symbol1 (implicit local storage) define
>> 

Re: [Kicad-developers] Ideas for a lightweight KiCad PDM

[Kevin Bortis]

On Tue, Dec 6, 2016 at 4:56 PM, Wayne Stambaugh  wrote:
> On 12/6/2016 2:22 AM, Kevin Bortis wrote:
>> Hi
>>
>> I try to figure out how it would be possible to create a central
>> storage for symbols, footprints and other design data. With central I
>> mean something like a design data server, where it is possible to
>> assign workflows with sign-offs, make proper versioning, assign order
>> numbers and define second sources. (PDM ultra light for KiCad).
>>
>> I currently work on some projects where we need to be sure that each
>> change on current products gets a proper review and documentation.
>> This always starts with a "Change Request", followed by a "Change
>> Order". To make sure everything is right, we currently use a very
>> expensive EDA suite with even more expensive PDM[2] software. Even if
>> this seems overkill for private projects, a little more lighter
>> version of this workflow makes sens for every company and even for
>> some bigger private projects. I helps to grantees some quality and
>> data integrity.
>
> I'm fine with this but I will resist anything that imposes a work flow
> on users.  Features like this tend to be very personal so I would
> request that it be implemented as a plugin or a python script.  Python
> scripting would require the schematic code to be swigged.  It's on our
> list of things to do but it wont happen until after the new schematic
> and symbol library file formats are completed.
>

Almost all logic would be implemented on server side inclusive
workflow stuff. Even when the topic sounds complex, the client does
not need a lot to support proper integration in such a tool. The
"normal" workflow would not be touched in any case. Implemented as a
plugin or not I see only the following things that would change:

  1. Additional KiCad settings for activating and setting "network
sources" for design data (One or more)

The following points should only be accessible if the settings in 1.
are modified and at least one "network source" is activated.

  2. Additional menu to verify data integrity of local cached data
  3. Additional context menu items for different actions

As you see, a normal user would in any case not be affected.
Disclaimer: The following statements are made from someone that has
just started to analyse the KiCad file formats and design decisions.
At some points I could be totally wrong or write just nonsense.

This sounds promising. I think you refer to the SWEET data format. [4]
I saw some commits and mailing list entries from 2011. Are there any
newer specifications or discussions that I have overseen? The drawing
in [3] does suggest, that there is even planned to add some remote
http library stuff. When I read the SWEET spec, there is a NAME_HINT
parameter specified that can be in different formats depending on the
context from which the SWEET part came from. To be exact, the file
format for all parts stored in a schematic and layout file need the
following features:

  * Object-ID that is the same in schematic and layout file. This must
be different to the designator and unique. It helps to quickly find
the same part in schematic and layout and allows neat features like
simultaneous highlighting, back and fort annotation and other great
stuff. This could be used to get around file based net list generation
and import to.
  * Definition of the origin or "where is the original stored". It
will be used to check the remote location if the part has a new
version or if the local cached version is unmodified. (Just hash and
compare S-expression blocks?)

Based on [4], i would then ask for two additional and optional
parameters: "objectid" and "origin". The "objectid" would be assigned
on the fly for every part placed on a schematic sheet. It will stay
the same, even if a property like the designator changes. The
parameter gets passed to the layout the same way other properties get
passed through. It would change however if the part gets copied. The
"origin" parameter would then be something like:
remote1/16fd2706-8baf-433b-82eb-8c7fada847da . If not htere the part
is not considered as versioned. The same two parameters are needed not
only for parts, but for schematics, footprints to. These two
properties get assigned during placement and do not preexist in the
library.

I see only one single glitch where it gets a little weird. The file
format for the footprint as descibed in [5] does directly specfiy the
3D-model whithin the footprint module itself. For network based
libraries, the 3D-model would be saved as a separate model (same level
like schematic symbol, footprint or a datasheet) in the database. This
means that after the footprint is created and 3D-model is placed, the
user would then first commit the 3D-model get an ID back for it (if
not already there) and then store only the reference to this object
along the at, scale and rotate position. I have not found that the
"model" property allows parameters like

Re: [Kicad-developers] Ideas for a lightweight KiCad PDM

[Wayne Stambaugh]

On 12/6/2016 2:22 AM, Kevin Bortis wrote:
> Hi
> 
> I try to figure out how it would be possible to create a central
> storage for symbols, footprints and other design data. With central I
> mean something like a design data server, where it is possible to
> assign workflows with sign-offs, make proper versioning, assign order
> numbers and define second sources. (PDM ultra light for KiCad).
> 
> I currently work on some projects where we need to be sure that each
> change on current products gets a proper review and documentation.
> This always starts with a "Change Request", followed by a "Change
> Order". To make sure everything is right, we currently use a very
> expensive EDA suite with even more expensive PDM[2] software. Even if
> this seems overkill for private projects, a little more lighter
> version of this workflow makes sens for every company and even for
> some bigger private projects. I helps to grantees some quality and
> data integrity.

I'm fine with this but I will resist anything that imposes a work flow
on users.  Features like this tend to be very personal so I would
request that it be implemented as a plugin or a python script.  Python
scripting would require the schematic code to be swigged.  It's on our
list of things to do but it wont happen until after the new schematic
and symbol library file formats are completed.

> 
> After some analysis how KiCad does things I see some little problems.
> The first of them is described in the following lines:
> 
> Currently the footprint and soon[1] symbols will be assigned with a
> "library nickname"/"symbol name" key. This is also true for the 3D
> model inside the  footprint file. At the moment this is the only
> reference to the library. If the component in the local file storage
> changes, the symbol/footprint gets changed immediately in the project
> and that is exactly what I want to avoid. To make a server based
> library, each symbol or footprint would need a unique identifier and
> some way to define the storage location. Say instead of define
> somelibrary/symbol1 (implicit local storage) define
> srv://remote1/16fd2706-8baf-433b-82eb-8c7fada847da (explicit remote
> storage) where remote1 is a configured remote storage location in the
> app. The fully qualified address of the symbol would then be something
> like https://myserver.local/lib/uuid/16fd2706-8baf-433b-82eb-8c7fada847da
> 
> A second possibility would be to introduce a new file (symbol.origin)
> in the project folder to keep track of every remote stored file.
> 
> symbol.orign:
> somelibrary/symbol1=remote1/16fd2706-8baf-433b-82eb-8c7fada847da
> somelibrary/footprint2=remote1/23fd2706-3baf-223b-82eb-ef7fada847ac

This issue will be pretty much be solved when the new schematic file
format is introduced because symbols will be embedded in the schematic
(unless you specifically externally link them which will be an option at
some point) so they will not change when the library symbol changes.  Of
course this introduces a new problem where the user will have to update
the symbols manually at least initially.  At some point in the future we
will provide a feature for comparing symbols in the schematic against
the symbol linked to the library and inform the user of any changes.

> 
> After retrival from the central storage, I would save the symbol as
> libs/somelibrary/symbol1 locally in the project tree. This would have
> almost no impact on the KiCad code I think. A command can be
> implemented to verifiy data integrity by getting file hashes of each
> component from the central server and checking if everything is in
> sync. A second command to check if a newer version of the symbol is
> available with the option to update it. To make this work, KiCad would
> need to track the origin of every 3D model, symbol, footprint...
> 
> I would appreciate if some one has some input on this topic.
> 
>   [1] https://lists.launchpad.net/kicad-developers/msg26901.html
>   [2] https://en.wikipedia.org/wiki/Product_data_management
> 
> Regards
> Kevin
> 
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
> 

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] Ideas for a lightweight KiCad PDM

[Clemens Koller]

Hello, Kevin!

I fully support your ideas. In a commercial environment, a proper PDM including 
version management and documentation is mandatory to be able to deliver a 
product where you can realize a traceability down to the date code of a 
component once your design is manufactured.

But since all projects / infrastructures / companies differ, it is propably not 
a good idea to hard code some workflow into the design tool (KiCad) and expect 
a one fits all approach.

I have sort of a wishlist which would allow me to interface KiCad to my 
existing infrastucture and solve some issues which cost me a lot of time:
- The library as well as all design data should be accessible (read-writeable) 
like a database (ideally: while KiCad is running).
- Everything (even every single connection in the schematics) has a version and 
a (modification) timestamp and is sortable by object, properties, version and 
timestamp.
- A copy (reuse) of fractions of a design should optionally include it's 
objects properties, constraints, versions and history.
- A diff and merge of parts or whole designs would improve versioning and 
design-reuse a lot.
* I would love to see forward/rewind buttons for libraries, schematics and 
layout similar to browsing through the git history of a software project 8-).

For performance reasons I believe it is not a good idea to implement design to 
library connectivity with links on a file-system level as it is done by bothy 
commercial products quite a lot. (One horribly expensive product seems to hold 
the whole design in an (optimized) database in memory and is able to replicate 
the data in real-time in between several workstations to feature "concurrent 
design entry" where several people/autorouters are able to work on one design 
at the same time.)

OT:
I am working a lot with databases (SQL based) and several external tools 
(self-written stuff + git) to manage my component library, the library 
generators, as well as my local component information / stock data. I am able 
to "link" a BOM written by the design tool with my component database (+ local 
stock data) and then with the i.e. SAP data (+ remote stock data) of my 
manufacturer to transfer my design to the assembly in an automated way. We 
implemented something like a closed-loop-verification process where I am able 
to read back the SAP data (maybe in the future through some remote database 
interface) into my database to verify that the data will arrive correctly at 
the machines. Sometimes, I'll need to update my local data when the assembly 
needs a change somewhere. Since everything is a moving target you want to 
freeze all the information for each manufacturing lot / product version.

These processes different a lot from one assembly house to the next one. So, 
there is a cookbook of scripts tailored to each manufacturers process with a 
lot of verification steps built in. There are usually no more .xls files 
envolved in the whole process where busy people tend to add typos manually 
without getting noticed.

Regards,

Clemens


On 2016-12-06 08:22, Kevin Bortis wrote:
> Hi
> 
> I try to figure out how it would be possible to create a central
> storage for symbols, footprints and other design data. With central I
> mean something like a design data server, where it is possible to
> assign workflows with sign-offs, make proper versioning, assign order
> numbers and define second sources. (PDM ultra light for KiCad).
> 
> I currently work on some projects where we need to be sure that each
> change on current products gets a proper review and documentation.
> This always starts with a "Change Request", followed by a "Change
> Order". To make sure everything is right, we currently use a very
> expensive EDA suite with even more expensive PDM[2] software. Even if
> this seems overkill for private projects, a little more lighter
> version of this workflow makes sens for every company and even for
> some bigger private projects. I helps to grantees some quality and
> data integrity.
> 
> After some analysis how KiCad does things I see some little problems.
> The first of them is described in the following lines:
> 
> Currently the footprint and soon[1] symbols will be assigned with a
> "library nickname"/"symbol name" key. This is also true for the 3D
> model inside the  footprint file. At the moment this is the only
> reference to the library. If the component in the local file storage
> changes, the symbol/footprint gets changed immediately in the project
> and that is exactly what I want to avoid. To make a server based
> library, each symbol or footprint would need a unique identifier and
> some way to define the storage location. Say instead of define
> somelibrary/symbol1 (implicit local storage) define
> srv://remote1/16fd2706-8baf-433b-82eb-8c7fada847da (explicit remote
> storage) where remote1 is a configured remote storage location in the
> app. The fully qualified address of the symbol would then be something
> 

[Kicad-developers] Ideas for a lightweight KiCad PDM

[Kevin Bortis]

Hi

I try to figure out how it would be possible to create a central
storage for symbols, footprints and other design data. With central I
mean something like a design data server, where it is possible to
assign workflows with sign-offs, make proper versioning, assign order
numbers and define second sources. (PDM ultra light for KiCad).

I currently work on some projects where we need to be sure that each
change on current products gets a proper review and documentation.
This always starts with a "Change Request", followed by a "Change
Order". To make sure everything is right, we currently use a very
expensive EDA suite with even more expensive PDM[2] software. Even if
this seems overkill for private projects, a little more lighter
version of this workflow makes sens for every company and even for
some bigger private projects. I helps to grantees some quality and
data integrity.

After some analysis how KiCad does things I see some little problems.
The first of them is described in the following lines:

Currently the footprint and soon[1] symbols will be assigned with a
"library nickname"/"symbol name" key. This is also true for the 3D
model inside the  footprint file. At the moment this is the only
reference to the library. If the component in the local file storage
changes, the symbol/footprint gets changed immediately in the project
and that is exactly what I want to avoid. To make a server based
library, each symbol or footprint would need a unique identifier and
some way to define the storage location. Say instead of define
somelibrary/symbol1 (implicit local storage) define
srv://remote1/16fd2706-8baf-433b-82eb-8c7fada847da (explicit remote
storage) where remote1 is a configured remote storage location in the
app. The fully qualified address of the symbol would then be something
like https://myserver.local/lib/uuid/16fd2706-8baf-433b-82eb-8c7fada847da

A second possibility would be to introduce a new file (symbol.origin)
in the project folder to keep track of every remote stored file.

symbol.orign:
somelibrary/symbol1=remote1/16fd2706-8baf-433b-82eb-8c7fada847da
somelibrary/footprint2=remote1/23fd2706-3baf-223b-82eb-ef7fada847ac

After retrival from the central storage, I would save the symbol as
libs/somelibrary/symbol1 locally in the project tree. This would have
almost no impact on the KiCad code I think. A command can be
implemented to verifiy data integrity by getting file hashes of each
component from the central server and checking if everything is in
sync. A second command to check if a newer version of the symbol is
available with the option to update it. To make this work, KiCad would
need to track the origin of every 3D model, symbol, footprint...

I would appreciate if some one has some input on this topic.

  [1] https://lists.launchpad.net/kicad-developers/msg26901.html
  [2] https://en.wikipedia.org/wiki/Product_data_management

Regards
Kevin

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp