| Accelerating
Feature Flow:
How Realization has integrated TOC and Lean to increase
throughput and responsiveness.
by
Ravi Shankar, VP of Software Development
Realization’s software
development department has been running all its projects
on Concerto since 2001. Cycle time for a normal release
was between three to four months and about six months
for major releases. Each release consisted of several
projects followed by final integration and testing.
Our due-date performance
was about 90 percent. However, scope was often changed
during execution to satisfy urgent new customer needs.
Our customers and our services department consistently
asked us to develop features and functionality at
a faster rate, both to support diverse environments
and to fully exploit the power of critical chain.
A quick analysis indicated
that, even though we were already very productive
compared to before 2001, we needed to increase our
throughput by another 50 percent.
Where to improve
After studying our entire
process, from feature request through development
and testing to deployment at the customer site, we
came to the following conclusions:
- Throughput
for our customers is not a version release, but
a fully working feature or functionality.
- Bundling
several bug fixes and enhancements into one release
was a major reason for throughput losses:
- With many features in
process, it was difficult to contain interruptions.
When there is a problem on one feature, one’s
tendency is to move to the next one rather than
solving that problem.
- Even if a feature was
completed early in the cycle, it had to wait until
the end of the release.
- Since customers do not
want to go through hefty upgrades frequently, it
takes six months or more from the time of release
for customers to use the features. Thus, a feature
developed by us got to the customer (converted into
real throughput) much later.
- Customer feedback on
new functionality came to us very late, affecting
our speed in making further enhancements.
- If a required feature
did not make it into a release, our customers had
to wait for the next release to get that feature.
Therefore, we always faced pressure to cram even
more features into a release, stretching the development
cycle still further.
 3.Another
major cause of loss in throughput was that technical
issues and bugs that came up during
design, coding and testing could not be resolved promptly:
- When a bug was found during
QA, developers had already moved on to other projects
and forgotten about the work they did on the feature
in question.
- Due to the high number
of features in process, issues were either not discovered
or not resolved quickly during design and development,
causing rework.
- Testers found themselves
jumping between features due to long turnaround
times in bug fixing. They would set up the test
system for a feature, find bugs, send them to development
for fixing, set up the system to test the next feature,
and when the bugs were fixed – redo the setup
for retesting. That is a substantial waste of both
time and capacity.
We realized that features
could be tested, fixed and completed much faster with
faster issue resolution. However, due to many features
in process, these features were getting stretched
out and throughput was lost.
Changes
to make
- Cut
the batching of features and bugs – Have more
frequent releases with fewer features in each release.
- Cut
the batching of tasks in development and testing
– Work only on a certain number of testable
units of throughput and complete them, before starting
others.
- Reduce
the number of concurrent features in QA.
Details of the changes
We decided upon an aggressive three-week release cycle.
Features that follow this cycle are bug fixes, and
minor to medium enhancements.
The number of
concurrent features in QA was limited to two. A feature
goes into QA when some or all code units are ready
for the feature and the end of the feature is fairly
clear. It is possible that more code units for that
feature are still being developed and will be tested
after the feature has already been introduced into
QA.
As soon as one of the concurrent
features is fully completed, another one is introduced
into QA. When testers are done testing and waiting
for bugs to be fixed by developers, they do infrastructure
work such as preparing test data and adding test cases
to our automated regressions. (In the past, testers
would move to testing the next feature.)
To ensure uninterrupted work
in QA, we maintain a buffer of three to four features
ready to go into QA. The release of work into design
and coding is paced based upon maintaining this buffer.
As a result, this buffer gives us flexibility in choosing
which features to introduce into the QA pipeline.
If the design and coding of a feature is delayed due
to unforeseen issues, the buffer makes sure that QA
still keeps working. The developer and lead resources
personnel assigned to the two features in QA are not
used for design and coding of features that are replenishing
the above buffer.
Emphasis during execution
is on getting the two features through testing as
correctly and quickly as possible without interruptions.
In execution, there is infrequently a major issue
that is going to delay a feature. In those rare situations,
we take the feature out of QA, reschedule it and introduce
a waiting feature.
Larger features and functionality
with multiple testable functional units may be broken
down to fit them into consecutive releases. Each such
testable unit is considered a feature from the release
control perspective. A bug reported by a customer
is also considered a feature since a bug fix can vary
from a day to a week or more of effort.
Management process
to support accelerated feature flow
- All
features are planned and pipelined in Concerto based
on the lead and QA capacity to ensure overall feasibility.
Pipeline Manager (that is me) establishes the sequence
in which features are release into execution.
- Each
feature is assigned a Feature Manager, who coordinates
the delivery of the feature with the developers
and QA Manager. Feature Managers are mostly lead
engineers, so they are also the ones who manage
task priorities.
- Project
plans are finalized just before design/coding is
going to start to replenish the ‘QA Buffer.’
- The
QA Manager pulls a feature out of the QA Buffer
upon completing a feature, informs the Feature Manager,
and assigns QA resources to start QA.
- The
Feature Manager is responsible for getting fixed
the bugs that are reported by QA.
Larger features that require
investigation, prototyping and major new development
are managed separately until it is determined that
they can be folded into this release process. Source-code
and release control changes to support this process
are not discussed here.
Expected Benefits
With more frequent releases, enhancements and bug
fixes get to our customers much faster. It also gives
them flexibility in choosing the version to upgrade,
based on the features they want.
Since fewer features go into
each release and the work-in-process is lower, features
do not get stretched out due to Interruptions, Parkinson’s
Law and scope creep. Lower work in process not only
gets the features done faster, it also enables better
quality control.
What about customer
upgrades?
Will our customers NEED to upgrade more often to take
advantage of our release cycle, creating an administrative
overhead?
As we see it, the frequency
of upgrades is still our customers’ choice.
Our goal currently is to give our customers and our
support department greater flexibility in selecting
the release for upgrade based on the customer’s
needs. If a customer finds that some very important
issue has been addressed, they can upgrade to the
release immediately instead of being forced to wait
in a longer release cycle.
From our perspective, even
a single customer upgrading to a release and using
a feature provides us quick feedback.
The above being said, with
fewer features in each release, training and testing
load on our customers is much lighter. We expect that
even our customers will want to upgrade more frequently.
But, batching of upgrades could be as unproductive
as batching in design, development and testing!
Please
let us know what you think, or if you have any suggestions.
Send an e-mail to FeatureFlow@realization.com.
back
to top  |
Kenneth
Lew joined Realization as scheduling engineer. In
a short-period, he has become a customers’ favorite
for quick troubleshooting of software issues and for
providing custom tools. Ken has a Bachelor’s
degree from UC-Davis and enjoys listening to music
and watching movies in his spare time. 
Anju
Kaul recently joined Realization as flow engineer
with significant technical, customer support and project
management experience. She has Bachelor’s and
Masters Degrees in Computer Science and Electrical
Engineering. She speaks five languages including Spanish
and Russian. Anju was vice-president of Hacienda Parks
toastmaster’s club, and in her spare time, enjoys
jazz music, golfing, reading and writing.
