Agile for Budgeting

Agile (Scrum in particular) is a framework for Product development, not just software development. It’s been successfully used for a wide variety of products, from medical devices to ad campaigns.

Let’s consider an organizational budget as another form of product, and explore if Agile might offer some advantages as an approach. Key among Agile values are embracing change; transparency; inspect and adapt.

Annual budgets are resistant to change, by their nature. While a sizeable portion of the budget might be necessarily fixed, not all of it need be. Some good-sized chunk of the overall budget can be marked as “adaptable”, being altered at some time-boxed frequency during the year, to best meet business priorities at that point in time, by a consistently organized team. Depending upon the organization, its culture and needs, the release of this could be quarterly or more often.

The overall annual budget tends to reflect a command and control culture in most organizations, even in those perceived otherwise. The budgeting typically is one pass, and then consolidated. Instead, try multiple passes, iterating, with transparency of plans between organizational units.

The funds for each item are implicitly the metric for business value; if there are other factors, then this needs to be explicitly stated, and ranking of business value made clear, for each budget item.

If an organization is going through a significant transformation, the budget not only should reflect this, but needs to adapt to changing circumstances along that transformational journey. The environment around the organization will change, and particularly for transformations, malleability is essential for success. Adaptive means agile, not the traditional approach.

note: originally posted to LinkedIn Pulse, Nov. 2015

Innovation and the Octopus

To create something innovative, one needs to look at things in a new way, going a quantum leap away from the well-trodden paths of the usual. To help stimulate creativity, try examining something unusual, something very different from our familiar world view.

Our way of viewing the world is that of an upright two legged, with two upper limbs and a hand at the end of each, having prehensile fingers.

Our companion animals usually are 4-legged, mostly with more acute sensory abilities than us, but overall, not too unlike ourselves and our view.

Our robotic creations tend to follow those structures and patterns, remaining close to the familiar.
For a very different perspective, consider the octopus and their marvelous abilities; a species millions of years older than our own.

Invertebrate: a large octopus can still squeeze through a 2 inch opening (or even smaller), presenting a challenge to keeping curious octopuses inside man-made containers or tanks. To date our robotic designs are vertebrate-centric; one might consider a soft robot, modeled after the octopus.

While almost all our cognitive neural capacity is centrally located in our brain, the octopus uses a more distributed model, with only about half centrally located, with the rest distributed to its limbs.

A user of tools and toys, one might contemplate what an octopus considers amusing or beautiful. What are the means and media of beauty and harmony for an octopus? How might that intersect with our own aesthetic concepts? Does the geometric pleasure of a Bach fugue resonate for an octopus?

Octopuses exhibit a wide range of behavioral patterns, perhaps falling into categories: are some more extroverted? How can one design a Meyers-Briggs type personality test for an octopus? For robots, how does one differentiate an introvert versus extrovert?

One might not readily associate the octopus with fashion and design, but some creative ideas might arise from contemplation of how a color-blind creature superbly camouflages more quickly than a chameleon. Could a chair be adaptive to the color or pattern of clothes worn by the person adorning it? Or have jewels adapt to nearby attire? (e.g., a modernized, Tiffany-class mood ring).

So much difference from our own way of being, our perceptions, should lead to some inspirations for innovation.

For those curious about the wonders of the octopus, try Sy Montgomery’s “Soul of an Octopus”. It’s a rich, clear and easy read, with the author’s tinge of wonder about the world.

Improving Requirements (Strengthening Waterfall)

One of the weaker phases in waterfall product development is the area of requirements.  Unclear or incorrect requirements are recognized as a major cause of grief on projects, if not outright disaster.

What is your feeling about the relationship between the requirements document and the ultimate product delivered?
How readily does the requirements document map to the required functionality?
Does it include a description or list of functionality that is out of scope? (“Thou shalt not…”)

In general, part of the problem is that there’s no real QA process on the requirements document, typically, only a user sign-off.  That’s an approval, not a QA process.  If the document is much over 10 pages, this is about as useful as handing someone a globe to use for direction to someone’s house for a holiday dinner.  You’re lucky if you even wind up in the right town, let alone the right neighborhood.

Typical waterfall environments have a QA function, usually for testing the finished product  Too often they only get involved late in development, often after all coding is allegedly done.  Why restrict their role so much?

Get them involved sooner, and involved with the analysts creating that requirements document. Involved, meaning interacting and talking directly to each other.  QA is designing the tests and should be able to identify unclear areas of the requirements, perhaps even missing parts. To flesh it out further, the QA group designs the tests based upon the requirements, and the business (not the BA who did the requirements) then reviews the test structures for completeness and accuracy, closing the feedback loop on the requirements document.  That process is independent of coding; it can start before any coding.

Another wobbly area is the clarity of the requirements for the developers.  Of course, clarity is only important when the document is actually read.

Another way to have some QA process for requirements is to have something similar to a peer review for coding, but in this case, instead of code, it is the requirements that are reviewed.

Come up with your variation of a QA process for requirements, one appropriate to the culture of your organization and resources available.

What’s Missing From Your SDLC

Having an SDLC (Software Development Life Cycle) documented is not to provide a minimum effort on development, or something to ignore, and keep writing programs as the mood strikes.  It’s part of an effort to follow best practices, to deliver high quality software on a repeated basis.  Even Agile methodologies can have an SDLC.

SDLC’s usually cover the core of the process of software creation, from initial requirements through design, coding, testing, integration, user acceptance, etc.  Some key elements are often overlooked, or considered not to be part of software development.  I’d like to suggest that you think about some areas that might improve the quality of the software you create if they are part of the SDLC.

Production support is not just about maintenance and fixing defects.  It’s about the ongoing improvement of what you delivered, preparation for the next release; part of the SDLC.  It’s about empowering the users, giving them the ability to fix their own problems when possible and reasonable; giving the right tools to the first level of support.  Not just documentation, an FAQ, or online Help, but items like creation of an error log and error trapping.  If the application has a database or data store, did you design a quality check on stored data, not just relying upon front end edit checking?

Does the application offer its users the opportunity to report problems, or give feedback about the interface they use?  What are the processes you envision for the ongoing improvement of the product?

Some SDLC’s include a Lessons Learned as part of closing out the development effort, project wrap-up.  The problem with that is it doesn’t do any good for the project in question, since that’s over, and seldom is examined either for the next release or by other projects.  Taking a page from the Agile approach of a retrospective for each iteration (sprint), why not do a Lessons Learned at each major milestone in the project, so the project can actually benefit while still in progress.

Security reviews should be part of the SDLC for all applications, not just web apps, not just for external facing ones.  How vulnerable is your application if another behind the firewall is compromised?  Security is part of the design process, not just something during the testing phase.

Security challenges are constantly changing, and some level of awareness needs to be part of the mindset of developers, not just IT security professionals.  Look towards organizations like OWASP (Open Web Application Security Project) as a resource.  Their Top Ten Security Risk list might startle some of your peers.

The Department of Homeland Security has some excellent resources, such as a National Vulnerability Database  as well as checklists and guidelines.  Look around their site.

Security is often a nightmare for developers, but there are people, organizations and tools to help you.

Look at your SDLC, think of the ongoing life of your applications, and what might enhance their quality.  The SDLC is an ongoing opportunity for process improvement.


Supporting Application Development

Methodologies or frameworks for the infrastructure to support Application Development (AD) is an interesting topic by itself, but one I’ve seldom seen mentioned or described, despite ITIL.  Even in companies that are following ITIL practices, the procedures are not often accompanied by feedback mechanisms to further improve things.

For example, do you trace incident or problem reports back to application change requests?  For major changes to application software, compare the post-implementation defect levels for different application groups.  Investigate the differences and consider ways to improve application development practices.

Many organizations believe that by designating support staff as level 1, 2 or 3, that they’ve established real procedures.  Other than “expected time to resolve” criteria, I’ve seldom heard of any real process analysis associated with those categories.  Nor have I heard of processes for a more organizationally flat structure.  A process needs to be more than assigning a severity level and contacting a software vendor.

What are your company’s infrastructure processes for supporting AD efforts?  What are the feedback mechanisms in those processes, such as periodic reviews?  If you were an infrastructure manager, how would you know how effective has been your support for AD, and where there may be perceived deficiencies?

Does each organization have a clear view of major plans and events for the next 3 to 6 months?  Does the AD area have any clues as to when systems is thinking of upgrading the version of the DBMS, or other essential software?

Hint to infrastructure:  if an AD team is pedal to the metal on delivering to the business, 2-3 weeks notice that you’re planning to upgrade to a new version of some key component is likely to be greeted by less than rabid enthusiasm.

Hint to the AD manager:  if you have not budgeted in time, resources and tooling (test scripts, etc.) for at least one version upgrade per year of the DBMS, and at least once every other year for the OS, some rude surprises may await you.

A tale of non-communication, with details altered out of sympathy:

Application development has a software product from a vendor, with much in-house enhancement.  In their published plans for the next quarter, AD had included an upgrade to a newer version of the product.  They knew that a newer version of the DBMS than currently used by the application would be needed for it, but since that newer version was already in use in-house by other applications, they were not concerned.

When the specific requirements became official requests to the infrastructure, the fun began.  The newer version of the DBMS required a newer version of the operating system than was currently used by that application.  And their servers were only marginal for the newer level of the OS; a hardware refresh was recommended.

The newer version of the OS not only brought along new run-time libraries, but used newer versions of compilers.  Even without any hardware refreshes, the regression testing alone for the DBMS and OS upgrades added significant time to the overall project.

Lesson learned:  frequent regular review of upcoming plans, including details as “DBMS version xxx needed.”  A review, meaning a meeting (even if virtual), with some communication, should be held.  Make sure the right people are talking to each other.

Rejuvenating the Infrastructure

The lack of agility of the infrastructure, especially the data center and systems programming/ administration is so obvious that the nascent DevOps movement has arisen to combat this perceived deficiency.  Much of that movement’s visible contribution has focused on automation and tools (e.g., Chef, Puppet, cfengine, Nagios) rather than on frameworks or processes.  Not surprising, given the talent for coding of many of the DevOps advocates.

There are other obstacles:

The Data Center values stability of the environment, and resists change; change is seen as a risk, not as any form of added value.  In general, perhaps especially for larger IT shops, other than changes due to bug fixes or security patches, the data center is isolated from positive aspects of application and environmental changes.  New metrics could provide a more flexible approach along the lines of increased Business Value (BV).  Some places, especially those along ITIL lines, may use a metric for risk as part of Change Control, but I have not heard of a corresponding one for enhanced BV.  That BV metric could be a 1 to 5 scale, or a set of numeric guidelines.

Much of the effort of the infrastructure is about process, with solutions well defined.  The challenges are more along the lines of:

–       Sheer scope of tasks, as the whole enterprise may be involved, with associated risk of that dimension.

–       Interdependencies

–       Complexity

–       Volume of work (especially with sharp peaks)

–       Lack of teams

For volume, automation and tools can help; one should also investigate frameworks like  Kanban, limiting the work in progress at any moment.

The lack of teams in infrastructure remains a source of concern for me.  Teams, behaving as teams, not just silos of individuals with common product-oriented expertise.  For example, a group of system administrators each with distinct separate sets of servers to manage, with no or little collaboration between members of the group, is not a team. The situation not only deprives the company and the participants of the benefits of a team approach, but reinforces the sense of isolation and endless repetition of upgrades and patches.

In my prior blog post on Skill-Centric Teams, a key point is that teams can be created across functional silos, basing them on the shared skills and challenges.  While there are distinct differences between a UNIX system administrator, the Windows counterpart, and their mainframe brethren, but there is so much in common, including, but not limited to:

–       Issues of change management

–       Communication with developers

–       Application development (AD) support methodologies or frameworks

Application development infrastructure support frameworks deserve a separate post.