In my career at Microsoft, I worked on many products which had to deliver updates (Ex: security fixes), as part of a servicing lifecycle. I have seen two different approaches where one utilizes a dedicated Servicing team and one requires the product team to also prepare servicing updates. There is benefit in both models and so let me give a comparison so you can consider your situation, to see which sounds best.
Let’s start by explaining the role of servicing (simplified view).
- Monitors customer reported issues (perhaps in partnership with a Customer Support team).
- Investigates and makes fix/won’t fix decisions and communications.
- Creates software updates for fixes.
- Releases software updates (perhaps in conjunction with an Operations team).
- Monitors software updates deployment success.
There is also the decision whether the updates will be created in a dedicated servicing code branch or directly out of a mainline branch. To keep this short, I will suggest that most servicing related updates/security fixes will come from a dedicated servicing branch as it simplifies expedited and scoped fixes. If the updates are not being made from a dedicated branch, then it will be hard to have a separate servicing organization as two teams with different purposes working on the same code in the same branch is a bad idea.
The biggest advantage of a dedicated servicing team is the ability to focus completely on customer issues. It is hard for the product team to both develop the next release while maintaining customer responsiveness for servicing fixes. There are a couple advantages to having a single product team that both develops the new product as well as responds to customer issues.
- Product Developers, who shipped the code, often have the best knowledge to investigate bugs in it.
- Making the servicing fixes helps the Product Developers learn from their mistakes and this helps avoid similar product issues in the future.
Best Practices for a Dedicated Servicing Organization
- A process must be established guaranteeing forward porting of the fixes as well as ensuring the product team learns from the issues. An effective mechanism is to automatically create a product tracking bug for every servicing fix and have that bug link back to the servicing bug and include details on the issue and code changes. This allows the product team to review and, through the act of forward porting the change, can drive learning.
- Servicing fixes need to be code reviewed in conjunction with the product team to ensure both teams agree on the fix decision and that the code changes make sense in the context of planned changes in the product under development.
- A lightweight process that allows the servicing team to recognize that customer reported issues have already been fixed in the next release is very helpful. This allows the servicing team to review the existing fix for backporting. I like to have the servicing engineer consult with her product team counterpart to discuss newly active bugs (facilitates the already fixed discovery) and then later for the formal code review of the changes that will be made to implement the fix they discussed.
These best practices help ensure that two teams remain in communication on what is fixed and how it has been fixed.
Best Practices for a Product Team Owning Servicing
- Management must set a tone that the customer fixes are the priority and resource scheduling must budget for this reserve. Ex: Servicing in an active fix area could reserve 15% of the resource’s scrum budget for re-occurring servicing task items and areas with few servicing fixes can reserve less, perhaps 5%.
- Management needs to track and adjust resourcing to ensure servicing remains the top priority while keeping the next product release on schedule.
It can be hard to meet servicing needs and not miss product deadlines when these share resources, so metrics, tracking, and monitoring needs to be maintained.
Other Servicing Best Practices
- When time permits, efficiency can be gained by fixing bugs in the product line branch and after it has been tested, backporting the fix to the servicing branch for final testing. This can also help during times of an overloaded servicing team.
- As the product development team finds issues with the not yet released product, high severity issues should raise a servicing tracking bug for backport consideration. Though the main process is to create servicing fixes in response to a customer input, exceptions will be made, and awareness is important.