Re: More on the Secretariat Statement of Work (SOW)
Firstly, I'll observe that this is outside the strict scope of the Secretariat SOW, since it covers the process cradle-to-grave, including WG, IESG, IANA and RFC Editor actions. Secondly, yes, dashboard metrics are a good idea, and are on the Tools team agenda, but often the devil is in the details and it's only by looking at specific cases of apparently stuck drafts that we can understand why things are moving slowly. Brian Spencer Dawkins wrote: Bernard/All, Ack on Bernard's note. I know that speed isn't the only thing that matters, but if we move slowly enough, the other stuff that matters won't matter. I'm remembering from previous discussions (sometime around the time of http://www3.ietf.org/proceedings/03mar/134.htm? or http://www3.ietf.org/proceedings/04mar/981.htm?) that the states we track in the ID tracker are sometimes overloaded, so it's hard to tell who has the token and exactly what is happening with the draft, and there are limits on what we've been able to do with metrics in the past. It's definitely worth thinking about this from a metrics perspective. Spencer In thinking through the Statement of Work (SoW), I think that an important component is to provide the IETF with sufficient information on how well the organization is performing. There are many metrics for that, but an important one is the time taken in various stages of the IETF process. Unfortunately, it is not clear to me that we are currently collecting this information in a form that makes it easy to analyze. We are also not analyzing the data on a regular basis, using it in a systematic effort to improve IETF performance (or at least to prevent it from deteriorating further). Researchers such as Tim Simcoe of the University of Toronto have studied metrics of IETF performance and have come to some interesting conclusions. For example, it appears that time from an initial -00 to RFC publication varies considerably by area, as well as by designation (Information, Experiemntal, Proposed). In the process of developing this research, Tim has also had to do significant work to adjust the data to make it suitable for analysis. My suggestion is that the IAOC needs to start thinking about what data and reports are needed to enable the IETF to measure and improve its performance. References -- Simcoe, T., Delays and de Jure Standards: What Caused the Slowdown in Internet Standards Development?, UC Berkeley Haas School of Business, April 2004. Simcoe, T., Standards Setting Committees, J.L Rotman School of Management, University of Toronto, Decmeber 2005. Available at: http://www.drizzle.com/~aboba/IAB/simcoe/ ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
More on the Secretariat Statement of Work (SOW)
In thinking through the Statement of Work (SoW), I think that an important component is to provide the IETF with sufficient information on how well the organization is performing. There are many metrics for that, but an important one is the time taken in various stages of the IETF process. Unfortunately, it is not clear to me that we are currently collecting this information in a form that makes it easy to analyze. We are also not analyzing the data on a regular basis, using it in a systematic effort to improve IETF performance (or at least to prevent it from deteriorating further). Researchers such as Tim Simcoe of the University of Toronto have studied metrics of IETF performance and have come to some interesting conclusions. For example, it appears that time from an initial -00 to RFC publication varies considerably by area, as well as by designation (Information, Experiemntal, Proposed). In the process of developing this research, Tim has also had to do significant work to adjust the data to make it suitable for analysis. My suggestion is that the IAOC needs to start thinking about what data and reports are needed to enable the IETF to measure and improve its performance. References -- Simcoe, T., Delays and de Jure Standards: What Caused the Slowdown in Internet Standards Development?, UC Berkeley Haas School of Business, April 2004. Simcoe, T., Standards Setting Committees, J.L Rotman School of Management, University of Toronto, Decmeber 2005. Available at: http://www.drizzle.com/~aboba/IAB/simcoe/ ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: More on the Secretariat Statement of Work (SOW)
Bernard/All, Ack on Bernard's note. I know that speed isn't the only thing that matters, but if we move slowly enough, the other stuff that matters won't matter. I'm remembering from previous discussions (sometime around the time of http://www3.ietf.org/proceedings/03mar/134.htm? or http://www3.ietf.org/proceedings/04mar/981.htm?) that the states we track in the ID tracker are sometimes overloaded, so it's hard to tell who has the token and exactly what is happening with the draft, and there are limits on what we've been able to do with metrics in the past. It's definitely worth thinking about this from a metrics perspective. Spencer In thinking through the Statement of Work (SoW), I think that an important component is to provide the IETF with sufficient information on how well the organization is performing. There are many metrics for that, but an important one is the time taken in various stages of the IETF process. Unfortunately, it is not clear to me that we are currently collecting this information in a form that makes it easy to analyze. We are also not analyzing the data on a regular basis, using it in a systematic effort to improve IETF performance (or at least to prevent it from deteriorating further). Researchers such as Tim Simcoe of the University of Toronto have studied metrics of IETF performance and have come to some interesting conclusions. For example, it appears that time from an initial -00 to RFC publication varies considerably by area, as well as by designation (Information, Experiemntal, Proposed). In the process of developing this research, Tim has also had to do significant work to adjust the data to make it suitable for analysis. My suggestion is that the IAOC needs to start thinking about what data and reports are needed to enable the IETF to measure and improve its performance. References -- Simcoe, T., Delays and de Jure Standards: What Caused the Slowdown in Internet Standards Development?, UC Berkeley Haas School of Business, April 2004. Simcoe, T., Standards Setting Committees, J.L Rotman School of Management, University of Toronto, Decmeber 2005. Available at: http://www.drizzle.com/~aboba/IAB/simcoe/ ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
