[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Thomas Monjalon Sent: Wednesday, October 8, 2014 4:57 PM To: Neil Horman Cc: dev at dpdk.org Subject: Re: [dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility Hi Neil, 2014-10-07 17:01, Neil Horman: > On Wed, Oct 01, 2014 at 02:59:40PM -0400, Neil Horman wrote: > > On Fri, Sep 26, 2014 at 10:45:49AM -0400, Neil Horman wrote: > > > On Fri, Sep 26, 2014 at 12:41:33PM +0200, Thomas Monjalon wrote: > > > > Hi Neil, > > > > > > > > 2014-09-24 14:19, Neil Horman: > > > > > Ping Thomas. I know you're busy, but I would like this to not > > > > > fall off anyones radar. You alluded to concerns regarding > > > > > what, for lack of a better term, ABI/API lockin. I had asked > > > > > you to enuumerate/elaborate on specifics, but never heard > > > > > back. Are there further specifics you wish to discuss, or are you > > > > > satisfied with the above answers? > > > > > > > > Sorry for not being very reactive on this thread. > > > > All this discussion is very interesting but it's really not the > > > > proper time to apply it. As you said, it requires an extra > > > > effort. I'm not saying it will never be integrated. I'm just > > > > saying that we cannot change everything at the same time. > > > > > > > > Let me sum up the situation. This community project has been > > > > very active for few months now. First, we learnt how to make > > > > some releases together and we are improving the process to be > > > > able to deliver a new major release every 4 months while having some > > > > good quality process. > > > > But these releases are still not complete because documentation > > > > is not integrated yet. Then developers should have a role in > > > > documentation updates. > > > > We also need to integrate and learn how to use more tools to be > > > > more efficient and improve quality. > > > > > > > > So the question is "when should we care about API compatibility"? > > > > And the answer is: ASAP, but not now. I feel next year is a better > > > > target. > > > > Because the most important priority is to move together at a > > > > pace which allow most of us to stay in the race. > > > > > > I'm sorry Thomas, I don't accept this. I asked you for details as > > > to your concerns regarding this patch series, and you've provided more > > > vague comments. > > > I need details to address > > > > > > You say it requires extra effort, you're right it does. Any > > > feature that you integreate requires some additional effort. How > > > is this patch any different from adding the acl library or any > > > other new API? Everything requires maintenence, thats how > > > software works. What specfically about this patch series makes the > > > effort insurmountable to you? > > > > > > You say you're improving your process. Great, this patch aids in > > > that process by ensuring backwards compatibility for a period of > > > time. Given that the API and ABI can still evolve within this > > > framework, as I've described, how is this patch series not a significant > > > step forward toward your goal of quality process. > > > > > > You say documentation isn't integrated. So, what does getting > > > documentation integrated have to do with this patch set, or any > > > other? I don't see you holding any other patches based on > > > documentation. Again, nothing in this series prevents evolution > > > of the API or ABI. If you're hope is to wait until everything is > > > perfect, then apply some control to the public facing API, and get it all > > > documented, none of thosse things will ever happen, I promise you. > > > > > > You say you also need to learn to use more tools to be more > > > efficient and improve quality. Great! Thats exactly what this > > > is. If we mandate even a short term commitment to ABI stability (1 > > > single relese worth of time), we will quickly identify what API's > > > change quickly and where we need to be cautious with our API > > > design. If you just assume that developers will get better of their own > > > volition, it will never happen. > > &
[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
Hi Neil, 2014-10-07 17:01, Neil Horman: > On Wed, Oct 01, 2014 at 02:59:40PM -0400, Neil Horman wrote: > > On Fri, Sep 26, 2014 at 10:45:49AM -0400, Neil Horman wrote: > > > On Fri, Sep 26, 2014 at 12:41:33PM +0200, Thomas Monjalon wrote: > > > > Hi Neil, > > > > > > > > 2014-09-24 14:19, Neil Horman: > > > > > Ping Thomas. I know you're busy, but I would like this to not fall > > > > > off anyones > > > > > radar. You alluded to concerns regarding what, for lack of a better > > > > > term, > > > > > ABI/API lockin. I had asked you to enuumerate/elaborate on > > > > > specifics, but never > > > > > heard back. Are there further specifics you wish to discuss, or are > > > > > you > > > > > satisfied with the above answers? > > > > > > > > Sorry for not being very reactive on this thread. > > > > All this discussion is very interesting but it's really not the proper > > > > time to apply it. As you said, it requires an extra effort. I'm not > > > > saying > > > > it will never be integrated. I'm just saying that we cannot change > > > > everything at the same time. > > > > > > > > Let me sum up the situation. This community project has been very active > > > > for few months now. First, we learnt how to make some releases together > > > > and we are improving the process to be able to deliver a new major > > > > release > > > > every 4 months while having some good quality process. > > > > But these releases are still not complete because documentation is not > > > > integrated yet. Then developers should have a role in documentation > > > > updates. > > > > We also need to integrate and learn how to use more tools to be more > > > > efficient and improve quality. > > > > > > > > So the question is "when should we care about API compatibility"? > > > > And the answer is: ASAP, but not now. I feel next year is a better > > > > target. > > > > Because the most important priority is to move together at a pace which > > > > allow most of us to stay in the race. > > > > > > I'm sorry Thomas, I don't accept this. I asked you for details as to your > > > concerns regarding this patch series, and you've provided more vague > > > comments. > > > I need details to address > > > > > > You say it requires extra effort, you're right it does. Any feature that > > > you > > > integreate requires some additional effort. How is this patch any > > > different > > > from adding the acl library or any other new API? Everything requires > > > maintenence, thats how software works. What specfically about this patch > > > series > > > makes the effort insurmountable to you? > > > > > > You say you're improving your process. Great, this patch aids in that > > > process > > > by ensuring backwards compatibility for a period of time. Given that the > > > API > > > and ABI can still evolve within this framework, as I've described, how is > > > this > > > patch series not a significant step forward toward your goal of quality > > > process. > > > > > > You say documentation isn't integrated. So, what does getting > > > documentation > > > integrated have to do with this patch set, or any other? I don't see you > > > holding any other patches based on documentation. Again, nothing in this > > > series > > > prevents evolution of the API or ABI. If you're hope is to wait until > > > everything is perfect, then apply some control to the public facing API, > > > and get > > > it all documented, none of thosse things will ever happen, I promise you. > > > > > > You say you also need to learn to use more tools to be more efficient and > > > improve quality. Great! Thats exactly what this is. If we mandate even > > > a short > > > term commitment to ABI stability (1 single relese worth of time), we will > > > quickly identify what API's change quickly and where we need to be > > > cautious with > > > our API design. If you just assume that developers will get better of > > > their own > > > volition, it will never happen. > > > > > > You say this should go in next year, but not now. When exactly? What > > > event do > > > you forsee occuring in the next 12-18 months that will change everything > > > such > > > that we can start supporing an ABI for more than just a few weeks at the > > > head of > > > the tree? > > > > > > To this end, I just did a quick search through the git history for dpdk > > > to look > > > at the histories of all the header files that are exposed via the makefile > > > SYMLINK command (given that that provides a list of header files that > > > applications can include, and embodies all the function symbols and data > > > types > > > applications have access to. > > > > > > There are 179 total commits in that list > > > Of those, a bit of spot checking suggests that about 10-15% of them > > > actually > > > change ABI, and many of those came from Bruce's rework of the mbuf > > > structure. > > > That about 17-20 instances over the last 2 years where an ABI u
[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
On Wed, Oct 08, 2014 at 05:57:29PM +0200, Thomas Monjalon wrote: > Hi Neil, > > 2014-10-07 17:01, Neil Horman: > > On Wed, Oct 01, 2014 at 02:59:40PM -0400, Neil Horman wrote: > > > On Fri, Sep 26, 2014 at 10:45:49AM -0400, Neil Horman wrote: > > > > On Fri, Sep 26, 2014 at 12:41:33PM +0200, Thomas Monjalon wrote: > > > > > Hi Neil, > > > > > > > > > > 2014-09-24 14:19, Neil Horman: > > > > > > Ping Thomas. I know you're busy, but I would like this to not fall > > > > > > off anyones > > > > > > radar. You alluded to concerns regarding what, for lack of a > > > > > > better term, > > > > > > ABI/API lockin. I had asked you to enuumerate/elaborate on > > > > > > specifics, but never > > > > > > heard back. Are there further specifics you wish to discuss, or > > > > > > are you > > > > > > satisfied with the above answers? > > > > > > > > > > Sorry for not being very reactive on this thread. > > > > > All this discussion is very interesting but it's really not the proper > > > > > time to apply it. As you said, it requires an extra effort. I'm not > > > > > saying > > > > > it will never be integrated. I'm just saying that we cannot change > > > > > everything at the same time. > > > > > > > > > > Let me sum up the situation. This community project has been very > > > > > active > > > > > for few months now. First, we learnt how to make some releases > > > > > together > > > > > and we are improving the process to be able to deliver a new major > > > > > release > > > > > every 4 months while having some good quality process. > > > > > But these releases are still not complete because documentation is not > > > > > integrated yet. Then developers should have a role in documentation > > > > > updates. > > > > > We also need to integrate and learn how to use more tools to be more > > > > > efficient and improve quality. > > > > > > > > > > So the question is "when should we care about API compatibility"? > > > > > And the answer is: ASAP, but not now. I feel next year is a better > > > > > target. > > > > > Because the most important priority is to move together at a pace > > > > > which > > > > > allow most of us to stay in the race. > > > > > > > > I'm sorry Thomas, I don't accept this. I asked you for details as to > > > > your > > > > concerns regarding this patch series, and you've provided more vague > > > > comments. > > > > I need details to address > > > > > > > > You say it requires extra effort, you're right it does. Any feature > > > > that you > > > > integreate requires some additional effort. How is this patch any > > > > different > > > > from adding the acl library or any other new API? Everything requires > > > > maintenence, thats how software works. What specfically about this > > > > patch series > > > > makes the effort insurmountable to you? > > > > > > > > You say you're improving your process. Great, this patch aids in that > > > > process > > > > by ensuring backwards compatibility for a period of time. Given that > > > > the API > > > > and ABI can still evolve within this framework, as I've described, how > > > > is this > > > > patch series not a significant step forward toward your goal of quality > > > > process. > > > > > > > > You say documentation isn't integrated. So, what does getting > > > > documentation > > > > integrated have to do with this patch set, or any other? I don't see > > > > you > > > > holding any other patches based on documentation. Again, nothing in > > > > this series > > > > prevents evolution of the API or ABI. If you're hope is to wait until > > > > everything is perfect, then apply some control to the public facing > > > > API, and get > > > > it all documented, none of thosse things will ever happen, I promise > > > > you. > > > > > > > > You say you also need to learn to use more tools to be more efficient > > > > and > > > > improve quality. Great! Thats exactly what this is. If we mandate > > > > even a short > > > > term commitment to ABI stability (1 single relese worth of time), we > > > > will > > > > quickly identify what API's change quickly and where we need to be > > > > cautious with > > > > our API design. If you just assume that developers will get better of > > > > their own > > > > volition, it will never happen. > > > > > > > > You say this should go in next year, but not now. When exactly? What > > > > event do > > > > you forsee occuring in the next 12-18 months that will change > > > > everything such > > > > that we can start supporing an ABI for more than just a few weeks at > > > > the head of > > > > the tree? > > > > > > > > To this end, I just did a quick search through the git history for dpdk > > > > to look > > > > at the histories of all the header files that are exposed via the > > > > makefile > > > > SYMLINK command (given that that provides a list of header files that > > > > applications can include, and embodies all the function symbols and > > >
[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
On Wed, Oct 01, 2014 at 02:59:40PM -0400, Neil Horman wrote: > On Fri, Sep 26, 2014 at 10:45:49AM -0400, Neil Horman wrote: > > On Fri, Sep 26, 2014 at 12:41:33PM +0200, Thomas Monjalon wrote: > > > Hi Neil, > > > > > > 2014-09-24 14:19, Neil Horman: > > > > Ping Thomas. I know you're busy, but I would like this to not fall off > > > > anyones > > > > radar. You alluded to concerns regarding what, for lack of a better > > > > term, > > > > ABI/API lockin. I had asked you to enuumerate/elaborate on specifics, > > > > but never > > > > heard back. Are there further specifics you wish to discuss, or are you > > > > satisfied with the above answers? > > > > > > Sorry for not being very reactive on this thread. > > > All this discussion is very interesting but it's really not the proper > > > time to apply it. As you said, it requires an extra effort. I'm not saying > > > it will never be integrated. I'm just saying that we cannot change > > > everything at the same time. > > > > > > Let me sum up the situation. This community project has been very active > > > for few months now. First, we learnt how to make some releases together > > > and we are improving the process to be able to deliver a new major release > > > every 4 months while having some good quality process. > > > But these releases are still not complete because documentation is not > > > integrated yet. Then developers should have a role in documentation > > > updates. > > > We also need to integrate and learn how to use more tools to be more > > > efficient and improve quality. > > > > > > So the question is "when should we care about API compatibility"? > > > And the answer is: ASAP, but not now. I feel next year is a better target. > > > Because the most important priority is to move together at a pace which > > > allow most of us to stay in the race. > > > > > > > > > I'm sorry Thomas, I don't accept this. I asked you for details as to your > > concerns regarding this patch series, and you've provided more vague > > comments. > > I need details to address > > > > You say it requires extra effort, you're right it does. Any feature that > > you > > integreate requires some additional effort. How is this patch any different > > from adding the acl library or any other new API? Everything requires > > maintenence, thats how software works. What specfically about this patch > > series > > makes the effort insurmountable to you? > > > > You say you're improving your process. Great, this patch aids in that > > process > > by ensuring backwards compatibility for a period of time. Given that the > > API > > and ABI can still evolve within this framework, as I've described, how is > > this > > patch series not a significant step forward toward your goal of quality > > process. > > > > You say documentation isn't integrated. So, what does getting documentation > > integrated have to do with this patch set, or any other? I don't see you > > holding any other patches based on documentation. Again, nothing in this > > series > > prevents evolution of the API or ABI. If you're hope is to wait until > > everything is perfect, then apply some control to the public facing API, > > and get > > it all documented, none of thosse things will ever happen, I promise you. > > > > You say you also need to learn to use more tools to be more efficient and > > improve quality. Great! Thats exactly what this is. If we mandate even a > > short > > term commitment to ABI stability (1 single relese worth of time), we will > > quickly identify what API's change quickly and where we need to be cautious > > with > > our API design. If you just assume that developers will get better of > > their own > > volition, it will never happen. > > > > You say this should go in next year, but not now. When exactly? What > > event do > > you forsee occuring in the next 12-18 months that will change everything > > such > > that we can start supporing an ABI for more than just a few weeks at the > > head of > > the tree? > > > > To this end, I just did a quick search through the git history for dpdk to > > look > > at the histories of all the header files that are exposed via the makefile > > SYMLINK command (given that that provides a list of header files that > > applications can include, and embodies all the function symbols and data > > types > > applications have access to. > > > > There are 179 total commits in that list > > Of those, a bit of spot checking suggests that about 10-15% of them actually > > change ABI, and many of those came from Bruce's rework of the mbuf > > structure. > > That about 17-20 instances over the last 2 years where an ABI update would > > have > > been needed. That seems pretty reasonable to me. Where exactly is your > > concern > > here? > > > > Neil > > > > Ping Thomas, I'd like to continue this debate to a conclusion. Could you > please > provide specific details and/or concerns that you ha
[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
On Fri, Sep 26, 2014 at 10:45:49AM -0400, Neil Horman wrote: > On Fri, Sep 26, 2014 at 12:41:33PM +0200, Thomas Monjalon wrote: > > Hi Neil, > > > > 2014-09-24 14:19, Neil Horman: > > > Ping Thomas. I know you're busy, but I would like this to not fall off > > > anyones > > > radar. You alluded to concerns regarding what, for lack of a better term, > > > ABI/API lockin. I had asked you to enuumerate/elaborate on specifics, > > > but never > > > heard back. Are there further specifics you wish to discuss, or are you > > > satisfied with the above answers? > > > > Sorry for not being very reactive on this thread. > > All this discussion is very interesting but it's really not the proper > > time to apply it. As you said, it requires an extra effort. I'm not saying > > it will never be integrated. I'm just saying that we cannot change > > everything at the same time. > > > > Let me sum up the situation. This community project has been very active > > for few months now. First, we learnt how to make some releases together > > and we are improving the process to be able to deliver a new major release > > every 4 months while having some good quality process. > > But these releases are still not complete because documentation is not > > integrated yet. Then developers should have a role in documentation updates. > > We also need to integrate and learn how to use more tools to be more > > efficient and improve quality. > > > > So the question is "when should we care about API compatibility"? > > And the answer is: ASAP, but not now. I feel next year is a better target. > > Because the most important priority is to move together at a pace which > > allow most of us to stay in the race. > > > > > I'm sorry Thomas, I don't accept this. I asked you for details as to your > concerns regarding this patch series, and you've provided more vague comments. > I need details to address > > You say it requires extra effort, you're right it does. Any feature that you > integreate requires some additional effort. How is this patch any different > from adding the acl library or any other new API? Everything requires > maintenence, thats how software works. What specfically about this patch > series > makes the effort insurmountable to you? > > You say you're improving your process. Great, this patch aids in that process > by ensuring backwards compatibility for a period of time. Given that the API > and ABI can still evolve within this framework, as I've described, how is this > patch series not a significant step forward toward your goal of quality > process. > > You say documentation isn't integrated. So, what does getting documentation > integrated have to do with this patch set, or any other? I don't see you > holding any other patches based on documentation. Again, nothing in this > series > prevents evolution of the API or ABI. If you're hope is to wait until > everything is perfect, then apply some control to the public facing API, and > get > it all documented, none of thosse things will ever happen, I promise you. > > You say you also need to learn to use more tools to be more efficient and > improve quality. Great! Thats exactly what this is. If we mandate even a > short > term commitment to ABI stability (1 single relese worth of time), we will > quickly identify what API's change quickly and where we need to be cautious > with > our API design. If you just assume that developers will get better of their > own > volition, it will never happen. > > You say this should go in next year, but not now. When exactly? What event > do > you forsee occuring in the next 12-18 months that will change everything such > that we can start supporing an ABI for more than just a few weeks at the head > of > the tree? > > To this end, I just did a quick search through the git history for dpdk to > look > at the histories of all the header files that are exposed via the makefile > SYMLINK command (given that that provides a list of header files that > applications can include, and embodies all the function symbols and data types > applications have access to. > > There are 179 total commits in that list > Of those, a bit of spot checking suggests that about 10-15% of them actually > change ABI, and many of those came from Bruce's rework of the mbuf structure. > That about 17-20 instances over the last 2 years where an ABI update would > have > been needed. That seems pretty reasonable to me. Where exactly is your > concern > here? > > Neil > Ping Thomas, I'd like to continue this debate to a conclusion. Could you please provide specific details and/or concerns that you have with this patch series? Thanks Neil > > -- > > Thomas > > >
[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
On Fri, Sep 26, 2014 at 03:02:55PM -0700, Stephen Hemminger wrote: > On Fri, 26 Sep 2014 10:45:49 -0400 > Neil Horman wrote: > > > On Fri, Sep 26, 2014 at 12:41:33PM +0200, Thomas Monjalon wrote: > > > Hi Neil, > > > > > > 2014-09-24 14:19, Neil Horman: > > > > Ping Thomas. I know you're busy, but I would like this to not fall off > > > > anyones > > > > radar. You alluded to concerns regarding what, for lack of a better > > > > term, > > > > ABI/API lockin. I had asked you to enuumerate/elaborate on specifics, > > > > but never > > > > heard back. Are there further specifics you wish to discuss, or are you > > > > satisfied with the above answers? > > > > > > Sorry for not being very reactive on this thread. > > > All this discussion is very interesting but it's really not the proper > > > time to apply it. As you said, it requires an extra effort. I'm not saying > > > it will never be integrated. I'm just saying that we cannot change > > > everything at the same time. > > > > > > Let me sum up the situation. This community project has been very active > > > for few months now. First, we learnt how to make some releases together > > > and we are improving the process to be able to deliver a new major release > > > every 4 months while having some good quality process. > > > But these releases are still not complete because documentation is not > > > integrated yet. Then developers should have a role in documentation > > > updates. > > > We also need to integrate and learn how to use more tools to be more > > > efficient and improve quality. > > > > > > So the question is "when should we care about API compatibility"? > > > And the answer is: ASAP, but not now. I feel next year is a better target. > > > Because the most important priority is to move together at a pace which > > > allow most of us to stay in the race. > > > > > > > > > I'm sorry Thomas, I don't accept this. I asked you for details as to your > > concerns regarding this patch series, and you've provided more vague > > comments. > > I need details to address > > > > You say it requires extra effort, you're right it does. Any feature that > > you > > integreate requires some additional effort. How is this patch any different > > from adding the acl library or any other new API? Everything requires > > maintenence, thats how software works. What specfically about this patch > > series > > makes the effort insurmountable to you? > > > > You say you're improving your process. Great, this patch aids in that > > process > > by ensuring backwards compatibility for a period of time. Given that the > > API > > and ABI can still evolve within this framework, as I've described, how is > > this > > patch series not a significant step forward toward your goal of quality > > process. > > > > You say documentation isn't integrated. So, what does getting documentation > > integrated have to do with this patch set, or any other? I don't see you > > holding any other patches based on documentation. Again, nothing in this > > series > > prevents evolution of the API or ABI. If you're hope is to wait until > > everything is perfect, then apply some control to the public facing API, > > and get > > it all documented, none of thosse things will ever happen, I promise you. > > > > You say you also need to learn to use more tools to be more efficient and > > improve quality. Great! Thats exactly what this is. If we mandate even a > > short > > term commitment to ABI stability (1 single relese worth of time), we will > > quickly identify what API's change quickly and where we need to be cautious > > with > > our API design. If you just assume that developers will get better of > > their own > > volition, it will never happen. > > > > You say this should go in next year, but not now. When exactly? What > > event do > > you forsee occuring in the next 12-18 months that will change everything > > such > > that we can start supporing an ABI for more than just a few weeks at the > > head of > > the tree? > > > > To this end, I just did a quick search through the git history for dpdk to > > look > > at the histories of all the header files that are exposed via the makefile > > SYMLINK command (given that that provides a list of header files that > > applications can include, and embodies all the function symbols and data > > types > > applications have access to. > > > > There are 179 total commits in that list > > Of those, a bit of spot checking suggests that about 10-15% of them actually > > change ABI, and many of those came from Bruce's rework of the mbuf > > structure. > > That about 17-20 instances over the last 2 years where an ABI update would > > have > > been needed. That seems pretty reasonable to me. Where exactly is your > > concern > > here? > > > > Neil > > Isn't ABI stablity a distro responsibility not a project responsibility? > I have lots more API/ABI changes, just been too busy trying to
[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
On Fri, 26 Sep 2014 10:45:49 -0400 Neil Horman wrote: > On Fri, Sep 26, 2014 at 12:41:33PM +0200, Thomas Monjalon wrote: > > Hi Neil, > > > > 2014-09-24 14:19, Neil Horman: > > > Ping Thomas. I know you're busy, but I would like this to not fall off > > > anyones > > > radar. You alluded to concerns regarding what, for lack of a better term, > > > ABI/API lockin. I had asked you to enuumerate/elaborate on specifics, > > > but never > > > heard back. Are there further specifics you wish to discuss, or are you > > > satisfied with the above answers? > > > > Sorry for not being very reactive on this thread. > > All this discussion is very interesting but it's really not the proper > > time to apply it. As you said, it requires an extra effort. I'm not saying > > it will never be integrated. I'm just saying that we cannot change > > everything at the same time. > > > > Let me sum up the situation. This community project has been very active > > for few months now. First, we learnt how to make some releases together > > and we are improving the process to be able to deliver a new major release > > every 4 months while having some good quality process. > > But these releases are still not complete because documentation is not > > integrated yet. Then developers should have a role in documentation updates. > > We also need to integrate and learn how to use more tools to be more > > efficient and improve quality. > > > > So the question is "when should we care about API compatibility"? > > And the answer is: ASAP, but not now. I feel next year is a better target. > > Because the most important priority is to move together at a pace which > > allow most of us to stay in the race. > > > > > I'm sorry Thomas, I don't accept this. I asked you for details as to your > concerns regarding this patch series, and you've provided more vague comments. > I need details to address > > You say it requires extra effort, you're right it does. Any feature that you > integreate requires some additional effort. How is this patch any different > from adding the acl library or any other new API? Everything requires > maintenence, thats how software works. What specfically about this patch > series > makes the effort insurmountable to you? > > You say you're improving your process. Great, this patch aids in that process > by ensuring backwards compatibility for a period of time. Given that the API > and ABI can still evolve within this framework, as I've described, how is this > patch series not a significant step forward toward your goal of quality > process. > > You say documentation isn't integrated. So, what does getting documentation > integrated have to do with this patch set, or any other? I don't see you > holding any other patches based on documentation. Again, nothing in this > series > prevents evolution of the API or ABI. If you're hope is to wait until > everything is perfect, then apply some control to the public facing API, and > get > it all documented, none of thosse things will ever happen, I promise you. > > You say you also need to learn to use more tools to be more efficient and > improve quality. Great! Thats exactly what this is. If we mandate even a > short > term commitment to ABI stability (1 single relese worth of time), we will > quickly identify what API's change quickly and where we need to be cautious > with > our API design. If you just assume that developers will get better of their > own > volition, it will never happen. > > You say this should go in next year, but not now. When exactly? What event > do > you forsee occuring in the next 12-18 months that will change everything such > that we can start supporing an ABI for more than just a few weeks at the head > of > the tree? > > To this end, I just did a quick search through the git history for dpdk to > look > at the histories of all the header files that are exposed via the makefile > SYMLINK command (given that that provides a list of header files that > applications can include, and embodies all the function symbols and data types > applications have access to. > > There are 179 total commits in that list > Of those, a bit of spot checking suggests that about 10-15% of them actually > change ABI, and many of those came from Bruce's rework of the mbuf structure. > That about 17-20 instances over the last 2 years where an ABI update would > have > been needed. That seems pretty reasonable to me. Where exactly is your > concern > here? > > Neil Isn't ABI stablity a distro responsibility not a project responsibility? I have lots more API/ABI changes, just been too busy trying to release a real product using DPDK to upstream all the changes.
[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
Hi Neil, 2014-09-24 14:19, Neil Horman: > Ping Thomas. I know you're busy, but I would like this to not fall off anyones > radar. You alluded to concerns regarding what, for lack of a better term, > ABI/API lockin. I had asked you to enuumerate/elaborate on specifics, but > never > heard back. Are there further specifics you wish to discuss, or are you > satisfied with the above answers? Sorry for not being very reactive on this thread. All this discussion is very interesting but it's really not the proper time to apply it. As you said, it requires an extra effort. I'm not saying it will never be integrated. I'm just saying that we cannot change everything at the same time. Let me sum up the situation. This community project has been very active for few months now. First, we learnt how to make some releases together and we are improving the process to be able to deliver a new major release every 4 months while having some good quality process. But these releases are still not complete because documentation is not integrated yet. Then developers should have a role in documentation updates. We also need to integrate and learn how to use more tools to be more efficient and improve quality. So the question is "when should we care about API compatibility"? And the answer is: ASAP, but not now. I feel next year is a better target. Because the most important priority is to move together at a pace which allow most of us to stay in the race. -- Thomas
[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
On Fri, Sep 26, 2014 at 12:41:33PM +0200, Thomas Monjalon wrote: > Hi Neil, > > 2014-09-24 14:19, Neil Horman: > > Ping Thomas. I know you're busy, but I would like this to not fall off > > anyones > > radar. You alluded to concerns regarding what, for lack of a better term, > > ABI/API lockin. I had asked you to enuumerate/elaborate on specifics, but > > never > > heard back. Are there further specifics you wish to discuss, or are you > > satisfied with the above answers? > > Sorry for not being very reactive on this thread. > All this discussion is very interesting but it's really not the proper > time to apply it. As you said, it requires an extra effort. I'm not saying > it will never be integrated. I'm just saying that we cannot change > everything at the same time. > > Let me sum up the situation. This community project has been very active > for few months now. First, we learnt how to make some releases together > and we are improving the process to be able to deliver a new major release > every 4 months while having some good quality process. > But these releases are still not complete because documentation is not > integrated yet. Then developers should have a role in documentation updates. > We also need to integrate and learn how to use more tools to be more > efficient and improve quality. > > So the question is "when should we care about API compatibility"? > And the answer is: ASAP, but not now. I feel next year is a better target. > Because the most important priority is to move together at a pace which > allow most of us to stay in the race. > I'm sorry Thomas, I don't accept this. I asked you for details as to your concerns regarding this patch series, and you've provided more vague comments. I need details to address You say it requires extra effort, you're right it does. Any feature that you integreate requires some additional effort. How is this patch any different from adding the acl library or any other new API? Everything requires maintenence, thats how software works. What specfically about this patch series makes the effort insurmountable to you? You say you're improving your process. Great, this patch aids in that process by ensuring backwards compatibility for a period of time. Given that the API and ABI can still evolve within this framework, as I've described, how is this patch series not a significant step forward toward your goal of quality process. You say documentation isn't integrated. So, what does getting documentation integrated have to do with this patch set, or any other? I don't see you holding any other patches based on documentation. Again, nothing in this series prevents evolution of the API or ABI. If you're hope is to wait until everything is perfect, then apply some control to the public facing API, and get it all documented, none of thosse things will ever happen, I promise you. You say you also need to learn to use more tools to be more efficient and improve quality. Great! Thats exactly what this is. If we mandate even a short term commitment to ABI stability (1 single relese worth of time), we will quickly identify what API's change quickly and where we need to be cautious with our API design. If you just assume that developers will get better of their own volition, it will never happen. You say this should go in next year, but not now. When exactly? What event do you forsee occuring in the next 12-18 months that will change everything such that we can start supporing an ABI for more than just a few weeks at the head of the tree? To this end, I just did a quick search through the git history for dpdk to look at the histories of all the header files that are exposed via the makefile SYMLINK command (given that that provides a list of header files that applications can include, and embodies all the function symbols and data types applications have access to. There are 179 total commits in that list Of those, a bit of spot checking suggests that about 10-15% of them actually change ABI, and many of those came from Bruce's rework of the mbuf structure. That about 17-20 instances over the last 2 years where an ABI update would have been needed. That seems pretty reasonable to me. Where exactly is your concern here? Neil > -- > Thomas >
[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
On Thu, Sep 18, 2014 at 03:14:01PM -0400, Neil Horman wrote: > On Thu, Sep 18, 2014 at 08:23:36PM +0200, Thomas Monjalon wrote: > > Hi Neil, > > > > 2014-09-15 15:23, Neil Horman: > > > The DPDK ABI develops and changes quickly, which makes it difficult for > > > applications to keep up with the latest version of the library, > > > especially when > > > it (the DPDK) is built as a set of shared objects, as applications may be > > > built > > > against an older version of the library. > > > > > > To mitigate this, this patch series introduces support for library and > > > symbol > > > versioning when the DPDK is built as a DSO. Specifically, it does 4 > > > things: > > > > > > 1) Adds initial support for library versioning. Each library now has a > > > version > > > map that explicitly calls out what symbols are exported to using > > > applications, > > > and assigns version(s) to them > > > > > > 2) Adds support macros so that when libraries create incompatible ABI's, > > > multiple versions may be supported so that applications linked against > > > older > > > DPDK releases can continue to function > > > > > > 3) Adds library soname versioning suffixes so that when ABI's must be > > > broken in > > > a fashion that requires a rebuild of older applications, they will break > > > at load > > > time, rather than cause unexpected issues at run time. > > > > > > 4) Adds documentation for ABI policy, and provides space to document > > > deprecated > > > ABI versions, so that applications might be warned of impending changes. > > > > > > With these elements in place the DPDK has some support to allow for the > > > extended > > > maintenence of older API's while still allowing the freedom to develop > > > new and > > > improved API's. > > > > > > Implementing this feature will require some additional effort on the part > > > of > > > developers and reviewers. When reviewing patches, must be checked against > > > existing exports to ensure that the function prototypes are not changing. > > > If > > > they are, the versioning macros must be used, and the library export map > > > should > > > be updated to reflect the new version of the function. > > > > > > When data structures change, if those structures are application > > > accessible, > > > apis that accept or return instances of those data structures should have > > > new > > > versions created so that users of the old data structure version might > > > co-exist > > > at the same time. > > > > Thanks for your efforts. > > But I feel this change has too many constraints for the current status of > > the DPDK. It's probably too early to adopt such policy. > > > I think you may be misunderstanding something. What constraints do you > beleive > that this patch imposes? Note it doesn't in any way prevent changes to the > ABI > of the DPDK, but rather gives us infrastructure to support multiple ABI > revisions at the same time, so that applications built against DPDK shared > libraries can continue to function properly at least for some time until we > decide to deprecate that ABI level. > > This is all based on the versioning strategy outlined here: > http://www.akkadia.org/drepper/dsohowto.pdf > > That may help clarify things for you. > > > By the way, this versioning doesn't cover structure changes. > No, it doesn't. No link-time mechanism does so. > > > How could it be managed? > Thats a subject that is open to discussion, but my initial thinking is that we > need to handle it on a case by case basis: > > * For minor updates, where allocation of a structure is done on the heap and > new > fields need to be added, appending them to the end of a structure and > providing > an initial value is sufficient. > > * For major changes, where fields need to be removed, or re-arranged, mostly > likely the API surfaces which accept or return those structures as > inputs/outputs will need to have new versions written to accept the new > version > of the structure, and internally we will have to support both formats for a > time > (according to the policy I documented, that is currently a single major > release). I.e. if you want to change struct foo, which is accepted as a > parameter for the function bar(struct foo *ptr), then for a release we would > need to create struct foo_v2 with the new format, map a new function foo_v2 to > the exported foo@@DPDK_1.(X+1), and internally make the foo functions > understand > both the origional and v2 versions of the structure. Then in DPDK release > 1.X+2, we can remove the old version after posting a deprecation notice with > version 1.(X+1) > > > Don't you think it would be more reliable if managed by packaging? > Solving this with packaging defeats the purpose of having shared libraries at > all. While packaging each version of the dpdk separately is possible stopgap > solution, in that it allows applications to link to differing versions of the > library independently, but that
[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
On Fri, Sep 19, 2014 at 07:18:36AM -0700, Venkatesan, Venky wrote: > On 9/18/2014 12:14 PM, Neil Horman wrote: > >On Thu, Sep 18, 2014 at 08:23:36PM +0200, Thomas Monjalon wrote: > >>Hi Neil, > >> > >>2014-09-15 15:23, Neil Horman: > >>>The DPDK ABI develops and changes quickly, which makes it difficult for > >>>applications to keep up with the latest version of the library, especially > >>>when > >>>it (the DPDK) is built as a set of shared objects, as applications may be > >>>built > >>>against an older version of the library. > >>> > >>>To mitigate this, this patch series introduces support for library and > >>>symbol > >>>versioning when the DPDK is built as a DSO. Specifically, it does 4 > >>>things: > >>> > >>>1) Adds initial support for library versioning. Each library now has a > >>>version > >>>map that explicitly calls out what symbols are exported to using > >>>applications, > >>>and assigns version(s) to them > >>> > >>>2) Adds support macros so that when libraries create incompatible ABI's, > >>>multiple versions may be supported so that applications linked against > >>>older > >>>DPDK releases can continue to function > >>> > >>>3) Adds library soname versioning suffixes so that when ABI's must be > >>>broken in > >>>a fashion that requires a rebuild of older applications, they will break > >>>at load > >>>time, rather than cause unexpected issues at run time. > >>> > >>>4) Adds documentation for ABI policy, and provides space to document > >>>deprecated > >>>ABI versions, so that applications might be warned of impending changes. > >>> > >>>With these elements in place the DPDK has some support to allow for the > >>>extended > >>>maintenence of older API's while still allowing the freedom to develop new > >>>and > >>>improved API's. > >>> > >>>Implementing this feature will require some additional effort on the part > >>>of > >>>developers and reviewers. When reviewing patches, must be checked against > >>>existing exports to ensure that the function prototypes are not changing. > >>>If > >>>they are, the versioning macros must be used, and the library export map > >>>should > >>>be updated to reflect the new version of the function. > >>> > >>>When data structures change, if those structures are application > >>>accessible, > >>>apis that accept or return instances of those data structures should have > >>>new > >>>versions created so that users of the old data structure version might > >>>co-exist > >>>at the same time. > >>Thanks for your efforts. > >>But I feel this change has too many constraints for the current status of > >>the DPDK. It's probably too early to adopt such policy. > >> > >I think you may be misunderstanding something. What constraints do you > >beleive > >that this patch imposes? Note it doesn't in any way prevent changes to the > >ABI > >of the DPDK, but rather gives us infrastructure to support multiple ABI > >revisions at the same time, so that applications built against DPDK shared > >libraries can continue to function properly at least for some time until we > >decide to deprecate that ABI level. > > > >This is all based on the versioning strategy outlined here: > >http://www.akkadia.org/drepper/dsohowto.pdf > > > >That may help clarify things for you. > > > >>By the way, this versioning doesn't cover structure changes. > >No, it doesn't. No link-time mechanism does so. > > > >>How could it be managed? > >Thats a subject that is open to discussion, but my initial thinking is that > >we > >need to handle it on a case by case basis: > > > >* For minor updates, where allocation of a structure is done on the heap and > >new > >fields need to be added, appending them to the end of a structure and > >providing > >an initial value is sufficient. > > > >* For major changes, where fields need to be removed, or re-arranged, mostly > >likely the API surfaces which accept or return those structures as > >inputs/outputs will need to have new versions written to accept the new > >version > >of the structure, and internally we will have to support both formats for a > >time > >(according to the policy I documented, that is currently a single major > >release). I.e. if you want to change struct foo, which is accepted as a > >parameter for the function bar(struct foo *ptr), then for a release we would > >need to create struct foo_v2 with the new format, map a new function foo_v2 > >to > >the exported foo@@DPDK_1.(X+1), and internally make the foo functions > >understand > >both the origional and v2 versions of the structure. Then in DPDK release > >1.X+2, we can remove the old version after posting a deprecation notice with > >version 1.(X+1) > > > >>Don't you think it would be more reliable if managed by packaging? > >Solving this with packaging defeats the purpose of having shared libraries at > >all. While packaging each version of the dpdk separately is possible stopgap > >solution, in that it allows applications to link to differing version
[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
> -Original Message- > From: Neil Horman [mailto:nhorman at tuxdriver.com] > Sent: Thursday, September 18, 2014 8:14 PM > To: Thomas Monjalon > Cc: dev at dpdk.org; Richardson, Bruce > Subject: Re: [PATCH 0/4] Add DSO symbol versioning to support backwards > compatibility > > On Thu, Sep 18, 2014 at 08:23:36PM +0200, Thomas Monjalon wrote: > > Hi Neil, > > > > 2014-09-15 15:23, Neil Horman: > > > The DPDK ABI develops and changes quickly, which makes it difficult for > > > applications to keep up with the latest version of the library, > > > especially when > > > it (the DPDK) is built as a set of shared objects, as applications may be > > > built > > > against an older version of the library. > > > > > > To mitigate this, this patch series introduces support for library and > > > symbol > > > versioning when the DPDK is built as a DSO. Specifically, it does 4 > > > things: > > > > > > 1) Adds initial support for library versioning. Each library now has a > > > version > > > map that explicitly calls out what symbols are exported to using > > > applications, > > > and assigns version(s) to them > > > > > > 2) Adds support macros so that when libraries create incompatible ABI's, > > > multiple versions may be supported so that applications linked against > > > older > > > DPDK releases can continue to function > > > > > > 3) Adds library soname versioning suffixes so that when ABI's must be > > > broken > in > > > a fashion that requires a rebuild of older applications, they will break > > > at load > > > time, rather than cause unexpected issues at run time. > > > > > > 4) Adds documentation for ABI policy, and provides space to document > deprecated > > > ABI versions, so that applications might be warned of impending changes. > > > > > > With these elements in place the DPDK has some support to allow for the > extended > > > maintenence of older API's while still allowing the freedom to develop new > and > > > improved API's. > > > > > > Implementing this feature will require some additional effort on the part > > > of > > > developers and reviewers. When reviewing patches, must be checked > against > > > existing exports to ensure that the function prototypes are not changing. > > > If > > > they are, the versioning macros must be used, and the library export map > should > > > be updated to reflect the new version of the function. > > > > > > When data structures change, if those structures are application > > > accessible, > > > apis that accept or return instances of those data structures should have > > > new > > > versions created so that users of the old data structure version might co- > exist > > > at the same time. > > > > Thanks for your efforts. > > But I feel this change has too many constraints for the current status of > > the DPDK. It's probably too early to adopt such policy. > > > I think you may be misunderstanding something. What constraints do you > beleive > that this patch imposes? Note it doesn't in any way prevent changes to the > ABI > of the DPDK, but rather gives us infrastructure to support multiple ABI > revisions at the same time, so that applications built against DPDK shared > libraries can continue to function properly at least for some time until we > decide to deprecate that ABI level. > I view all this as a positive step. I consider backward compatibility as something that should always be encouraged, and I agree with Neil that this should allow us to guarantee compatibility for our customers while still having a path open to us to change things if we really need to. > This is all based on the versioning strategy outlined here: > http://www.akkadia.org/drepper/dsohowto.pdf > > That may help clarify things for you. > > > By the way, this versioning doesn't cover structure changes. > No, it doesn't. No link-time mechanism does so. > > > How could it be managed? > Thats a subject that is open to discussion, but my initial thinking is that we > need to handle it on a case by case basis: > > * For minor updates, where allocation of a structure is done on the heap and > new > fields need to be added, appending them to the end of a structure and > providing > an initial value is sufficient. > > * For major changes, where fields need to be removed, or re-arranged, mostly > likely the API surfaces which accept or return those structures as > inputs/outputs will need to have new versions written to accept the new > version > of the structure, and internally we will have to support both formats for a > time > (according to the policy I documented, that is currently a single major > release). I.e. if you want to change struct foo, which is accepted as a > parameter for the function bar(struct foo *ptr), then for a release we would > need to create struct foo_v2 with the new format, map a new function foo_v2 > to > the exported foo@@DPDK_1.(X+1), and internally make the foo functions > understand > both the origional and v2 versions of
[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
On 9/18/2014 12:14 PM, Neil Horman wrote: > On Thu, Sep 18, 2014 at 08:23:36PM +0200, Thomas Monjalon wrote: >> Hi Neil, >> >> 2014-09-15 15:23, Neil Horman: >>> The DPDK ABI develops and changes quickly, which makes it difficult for >>> applications to keep up with the latest version of the library, especially >>> when >>> it (the DPDK) is built as a set of shared objects, as applications may be >>> built >>> against an older version of the library. >>> >>> To mitigate this, this patch series introduces support for library and >>> symbol >>> versioning when the DPDK is built as a DSO. Specifically, it does 4 things: >>> >>> 1) Adds initial support for library versioning. Each library now has a >>> version >>> map that explicitly calls out what symbols are exported to using >>> applications, >>> and assigns version(s) to them >>> >>> 2) Adds support macros so that when libraries create incompatible ABI's, >>> multiple versions may be supported so that applications linked against older >>> DPDK releases can continue to function >>> >>> 3) Adds library soname versioning suffixes so that when ABI's must be >>> broken in >>> a fashion that requires a rebuild of older applications, they will break at >>> load >>> time, rather than cause unexpected issues at run time. >>> >>> 4) Adds documentation for ABI policy, and provides space to document >>> deprecated >>> ABI versions, so that applications might be warned of impending changes. >>> >>> With these elements in place the DPDK has some support to allow for the >>> extended >>> maintenence of older API's while still allowing the freedom to develop new >>> and >>> improved API's. >>> >>> Implementing this feature will require some additional effort on the part of >>> developers and reviewers. When reviewing patches, must be checked against >>> existing exports to ensure that the function prototypes are not changing. >>> If >>> they are, the versioning macros must be used, and the library export map >>> should >>> be updated to reflect the new version of the function. >>> >>> When data structures change, if those structures are application accessible, >>> apis that accept or return instances of those data structures should have >>> new >>> versions created so that users of the old data structure version might >>> co-exist >>> at the same time. >> Thanks for your efforts. >> But I feel this change has too many constraints for the current status of >> the DPDK. It's probably too early to adopt such policy. >> > I think you may be misunderstanding something. What constraints do you > beleive > that this patch imposes? Note it doesn't in any way prevent changes to the > ABI > of the DPDK, but rather gives us infrastructure to support multiple ABI > revisions at the same time, so that applications built against DPDK shared > libraries can continue to function properly at least for some time until we > decide to deprecate that ABI level. > > This is all based on the versioning strategy outlined here: > http://www.akkadia.org/drepper/dsohowto.pdf > > That may help clarify things for you. > >> By the way, this versioning doesn't cover structure changes. > No, it doesn't. No link-time mechanism does so. > >> How could it be managed? > Thats a subject that is open to discussion, but my initial thinking is that we > need to handle it on a case by case basis: > > * For minor updates, where allocation of a structure is done on the heap and > new > fields need to be added, appending them to the end of a structure and > providing > an initial value is sufficient. > > * For major changes, where fields need to be removed, or re-arranged, mostly > likely the API surfaces which accept or return those structures as > inputs/outputs will need to have new versions written to accept the new > version > of the structure, and internally we will have to support both formats for a > time > (according to the policy I documented, that is currently a single major > release). I.e. if you want to change struct foo, which is accepted as a > parameter for the function bar(struct foo *ptr), then for a release we would > need to create struct foo_v2 with the new format, map a new function foo_v2 to > the exported foo@@DPDK_1.(X+1), and internally make the foo functions > understand > both the origional and v2 versions of the structure. Then in DPDK release > 1.X+2, we can remove the old version after posting a deprecation notice with > version 1.(X+1) > >> Don't you think it would be more reliable if managed by packaging? > Solving this with packaging defeats the purpose of having shared libraries at > all. While packaging each version of the dpdk separately is possible stopgap > solution, in that it allows applications to link to differing versions of the > library independently, but that negates any expectation of timely bugfixes for > any given version of the DPDK. That is to say, if you package things this > way, > and wind up with several parallel versions of t
[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
Hi Neil, 2014-09-15 15:23, Neil Horman: > The DPDK ABI develops and changes quickly, which makes it difficult for > applications to keep up with the latest version of the library, especially > when > it (the DPDK) is built as a set of shared objects, as applications may be > built > against an older version of the library. > > To mitigate this, this patch series introduces support for library and symbol > versioning when the DPDK is built as a DSO. Specifically, it does 4 things: > > 1) Adds initial support for library versioning. Each library now has a > version > map that explicitly calls out what symbols are exported to using applications, > and assigns version(s) to them > > 2) Adds support macros so that when libraries create incompatible ABI's, > multiple versions may be supported so that applications linked against older > DPDK releases can continue to function > > 3) Adds library soname versioning suffixes so that when ABI's must be broken > in > a fashion that requires a rebuild of older applications, they will break at > load > time, rather than cause unexpected issues at run time. > > 4) Adds documentation for ABI policy, and provides space to document > deprecated > ABI versions, so that applications might be warned of impending changes. > > With these elements in place the DPDK has some support to allow for the > extended > maintenence of older API's while still allowing the freedom to develop new and > improved API's. > > Implementing this feature will require some additional effort on the part of > developers and reviewers. When reviewing patches, must be checked against > existing exports to ensure that the function prototypes are not changing. If > they are, the versioning macros must be used, and the library export map > should > be updated to reflect the new version of the function. > > When data structures change, if those structures are application accessible, > apis that accept or return instances of those data structures should have new > versions created so that users of the old data structure version might > co-exist > at the same time. Thanks for your efforts. But I feel this change has too many constraints for the current status of the DPDK. It's probably too early to adopt such policy. By the way, this versioning doesn't cover structure changes. How could it be managed? Don't you think it would be more reliable if managed by packaging? Thank you for opening this discussion with a constructive proposal. Let's check it later on once structures will be more stable since performance is the most critical target. -- Thomas
[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
On Thu, Sep 18, 2014 at 08:23:36PM +0200, Thomas Monjalon wrote: > Hi Neil, > > 2014-09-15 15:23, Neil Horman: > > The DPDK ABI develops and changes quickly, which makes it difficult for > > applications to keep up with the latest version of the library, especially > > when > > it (the DPDK) is built as a set of shared objects, as applications may be > > built > > against an older version of the library. > > > > To mitigate this, this patch series introduces support for library and > > symbol > > versioning when the DPDK is built as a DSO. Specifically, it does 4 things: > > > > 1) Adds initial support for library versioning. Each library now has a > > version > > map that explicitly calls out what symbols are exported to using > > applications, > > and assigns version(s) to them > > > > 2) Adds support macros so that when libraries create incompatible ABI's, > > multiple versions may be supported so that applications linked against older > > DPDK releases can continue to function > > > > 3) Adds library soname versioning suffixes so that when ABI's must be > > broken in > > a fashion that requires a rebuild of older applications, they will break at > > load > > time, rather than cause unexpected issues at run time. > > > > 4) Adds documentation for ABI policy, and provides space to document > > deprecated > > ABI versions, so that applications might be warned of impending changes. > > > > With these elements in place the DPDK has some support to allow for the > > extended > > maintenence of older API's while still allowing the freedom to develop new > > and > > improved API's. > > > > Implementing this feature will require some additional effort on the part of > > developers and reviewers. When reviewing patches, must be checked against > > existing exports to ensure that the function prototypes are not changing. > > If > > they are, the versioning macros must be used, and the library export map > > should > > be updated to reflect the new version of the function. > > > > When data structures change, if those structures are application accessible, > > apis that accept or return instances of those data structures should have > > new > > versions created so that users of the old data structure version might > > co-exist > > at the same time. > > Thanks for your efforts. > But I feel this change has too many constraints for the current status of > the DPDK. It's probably too early to adopt such policy. > I think you may be misunderstanding something. What constraints do you beleive that this patch imposes? Note it doesn't in any way prevent changes to the ABI of the DPDK, but rather gives us infrastructure to support multiple ABI revisions at the same time, so that applications built against DPDK shared libraries can continue to function properly at least for some time until we decide to deprecate that ABI level. This is all based on the versioning strategy outlined here: http://www.akkadia.org/drepper/dsohowto.pdf That may help clarify things for you. > By the way, this versioning doesn't cover structure changes. No, it doesn't. No link-time mechanism does so. > How could it be managed? Thats a subject that is open to discussion, but my initial thinking is that we need to handle it on a case by case basis: * For minor updates, where allocation of a structure is done on the heap and new fields need to be added, appending them to the end of a structure and providing an initial value is sufficient. * For major changes, where fields need to be removed, or re-arranged, mostly likely the API surfaces which accept or return those structures as inputs/outputs will need to have new versions written to accept the new version of the structure, and internally we will have to support both formats for a time (according to the policy I documented, that is currently a single major release). I.e. if you want to change struct foo, which is accepted as a parameter for the function bar(struct foo *ptr), then for a release we would need to create struct foo_v2 with the new format, map a new function foo_v2 to the exported foo@@DPDK_1.(X+1), and internally make the foo functions understand both the origional and v2 versions of the structure. Then in DPDK release 1.X+2, we can remove the old version after posting a deprecation notice with version 1.(X+1) > Don't you think it would be more reliable if managed by packaging? Solving this with packaging defeats the purpose of having shared libraries at all. While packaging each version of the dpdk separately is possible stopgap solution, in that it allows applications to link to differing versions of the library independently, but that negates any expectation of timely bugfixes for any given version of the DPDK. That is to say, if you package things this way, and wind up with several parallel versions of the same package, and for any bugfix that comes out upstream, the packager then has the responsibility to adapt that fix to each package.
[dpdk-dev] [PATCH 0/4] Add DSO symbol versioning to support backwards compatibility
The DPDK ABI develops and changes quickly, which makes it difficult for applications to keep up with the latest version of the library, especially when it (the DPDK) is built as a set of shared objects, as applications may be built against an older version of the library. To mitigate this, this patch series introduces support for library and symbol versioning when the DPDK is built as a DSO. Specifically, it does 4 things: 1) Adds initial support for library versioning. Each library now has a version map that explicitly calls out what symbols are exported to using applications, and assigns version(s) to them 2) Adds support macros so that when libraries create incompatible ABI's, multiple versions may be supported so that applications linked against older DPDK releases can continue to function 3) Adds library soname versioning suffixes so that when ABI's must be broken in a fashion that requires a rebuild of older applications, they will break at load time, rather than cause unexpected issues at run time. 4) Adds documentation for ABI policy, and provides space to document deprecated ABI versions, so that applications might be warned of impending changes. With these elements in place the DPDK has some support to allow for the extended maintenence of older API's while still allowing the freedom to develop new and improved API's. Implementing this feature will require some additional effort on the part of developers and reviewers. When reviewing patches, must be checked against existing exports to ensure that the function prototypes are not changing. If they are, the versioning macros must be used, and the library export map should be updated to reflect the new version of the function. When data structures change, if those structures are application accessible, apis that accept or return instances of those data structures should have new versions created so that users of the old data structure version might co-exist at the same time. Signed-off-by: Neil Horman CC: Thomas Monjalon CC: "Richardson, Bruce"
