A group who shares ideas and experiences employing innovative acquisition practices, collaborative methods and use of Web2.0 technologies to transform federal acquisition.
Improving the Requirements Development Process
May 28, 2009 at 1:07 am #72684
We all know one of our biggest challenges in federal acquisition is defining requirements – meaning, defining requirements tightly enough so that we convey our needs, get good proposals from industry, eliminate cost overruns, scope creep and change orders, and ultimately get the outcome and results intended. So what makes defining requirements such a challenge for us? Is it the constrained timelines under which we are operating? Is it the ability to write a good requirements document that allows enough flexibility to take advantage of new technology during the development process? Are our program managers trained well enough in this area? Is it a resource shortage in general that forces us to take short cuts only to eventually prolong the timeline and costs?
Would making the requirements development process more collaborative and transparent help? For instance, if we opened up the process to industry in a semi-public forum, such as using a wiki to identify requirements and ask for input publicly – are we ready for this in government? What about the industry – are you ready to comment, offer suggestions and ask questions publicly? Would it compromise your competitive advantage or help?
May 28, 2009 at 1:53 am #72736
the more time we waste on developing ‘better’ requirements is time that could be spent on software code. At most you’ve got 6 months before all the assumptions we make in the process are aver taken by new hardware, software languages, etc.
We need to get away from treating acquisitions of IT systems like buying a tank or a TV. These systems are never done and continually dynamically evolving
Better to hire a good cadre of developers to build, extend and evolve systems vs. thinking it will be delivered at the end of a long and tortured process.
also better to fail on a small scale faster, than one big large failure at the end.
May 28, 2009 at 2:30 am #72734
That is an interesting and insightful comment. Are you saying that technology moves so quickly that we have no time to develop requirements? Even when acquiring tanks and TVs the IT componnents are so critical that your comment would apply..
I am not sure I agree with the notion of “hiring” (do we mean contracting?) with a set of software developers. That seems very expensve and leaderless. I would prefer to rely on an in-house cadre of system engineers to attempt to define requirements and perhaps to try to project the future state.
In pursuit of that goal, I believe the topic turns on the best approach to enlist expertise from industry.
May 28, 2009 at 3:09 am #72732
Kim Patrick KobzaParticipant
This is a key point and again, admitting my bias, requirements development (which is basically innovation and design) could be a collaborative and open process. This would provide the government with best of breed and innovative ideas and level the playing field. Also, a key point is that the best solutions are most often the combination of multiple memes building on themselves – not merely one idea. Why not enlist the assets of citizen, academic, and vendor networks to assist in idea generation and technical writing? Perfect area for social collaboration.
For some, but maybe not all requirements, social collaboration also provides the ability to reach into not simply the vendor community, but academia and pools of skilled retirees. One of the statistics that is relevant to the use of social collaboration in scope and definitional processes is this metric: 9 out of 10 new products fail. What if we reduced the failure rate by 10%? Even small reductions in failure rates often result in an increase in internal rate of return of up to 20%, Social collaboration properly applied can do that.
The other benefit of social collaboration in scoping – especially in government is compression of cycle times. This is important because technology especially has a limited useful life-cycle – often less than 24 months. For long scoping cycles government designs lead to functional obsolescence, simply as a function of time required in systems design and then implementation. We end up trying to solve today’s problems with yesterday’s state of the art solutions.
Functional obsolescence is most pronounced in communications technologies, although, it could be argued that it also exists in transactional systems. Standardization of rules across government would solve a great deal of this. But also, reordering unstructured communications structures to position them at early stages of design and decision making processes would go a long way towards early issue identification and collaborative solutions – all guided by program managers.
On a competitive level, an open process would level the playing field. Today, individual vendors essentially make their case, roll those ideas into an RFP/RFQ and predispose awards. You might find that a more open process would surface the small innovative and emerging companies willing to compete on the basis of ideas. The larger more centralized companies that often rely on their ability to aggregate assets to navigate myriad rules, and access to government officials, would have to compete on the basis of value added. This would have a secondary benefit of dramatically lessening transactions costs to government that are characteristic of highly complex processes.
Government should reward those capable of solving complex problems with simple solutions rather than those who take simple problems and make them complex.
On methodology of participation, there are many types of network services that will be appropriate for different types of business problems inherent in the process. Wikis might be appropriate to chronicle institutional memory for a design process, but 2 or 3 phased processes for structured communications and social collaboration might be better in for instance, a process that is built on alternative analysis. Those have different needs.
There are positives and negatives to each. Idea submission can also be closed if need be with submissions treated as confidential.
The key policy and practical question is whether gov would like to use its networks to provide accreditation and relevancy assessments to different possible solutions. Forward thinkers, like Yochai Benkler, Wealth of Networks, ascribe to this view and see public involvement in review and accreditation as a social good. Again, comes back to a policy decision in each case.
The other obvious benefit to program managers is free learning and education. This probably could work in both directions. The possibility of openness in design, scope and requirements is truly transformative.
May 28, 2009 at 12:14 pm #72730
Peter G. TuttleParticipant
I think industry would be reluctant to provide meaningful requirements input in a public forum without some degree of anonymity. I’ve been involved in too way many open forums where industry says virtually nothing (industry days, pre-solicitation conferences, RFI’s) because of competitive concerns. The goal for the companies I’ve worked for has always been to get in and see the government in a face-to-face meeting to discuss or “shape” the requirements.
Although I understand and agree with industry’s rationale for caution, I do see constrained communications as a barrier to an effective requirements formulation process.
Can we have more dialog on John’s and Gwynne’s points? I agree that the requirements development process can be convoluted and agonizing, but we’ve also all seen situations where the process is “streamlined” to the point where requirements are ambiguous and non-reflective of what the government is actually trying to achieve. Where do we draw the line? Obviously, the government needs to communicate its needs in a manner that they are understood by industry.
May 28, 2009 at 2:34 pm #72728
Devin B. HedgeParticipant
Has anyone here read up on Agile Management and Agile Software Development processes? Basically, one of the first principles is that the only constant is change; specifically, changing technology and requirements. This was illustrated in “The Mythical Man Month”in 1975, in “Death March”, “Facts and Fallacies of Software Engineering (Agile Software Development)”, “Rapid Development: Taming Wild Software Schedules”, and on and on.
We often confuse capabilities with requirements, and requirements with user expectations. I’ve rarely seen changes in what capabilities the government is asked/legislatively required to deliver. I do see changes in the requirements from one administration to another, thus necessitating changes to the system and software requirements. I do see changes in user expectations with EVERY iteration of a requirements document. Why? Because we have a lot of bright people in the country and are thinking up new and better ways to do things.
Here is the rub: the Federal Acquisition Lifecycle doesn’t allow for this level of innovation. Technology is moving at an exponentially increasing rate faster than Moore’s Law. Remember, Moore’s Law only applies to hardware. Software in turn, has not progressed at the same rate as hardware as evidenced by the fact that most computers have more processing capabilities than software using the processors. (Refer to VMWare and Forrester Research’s hardware utilization survey from 2006). Software has lagged behind hardware largely because it takes time to create and people are still figuring out how to use the tools: people are what’s holding software back. This is changing rapidly as we see the generational shift in the workforce. As each generation enters/exits the workforce the barriers to software creation drop and new ways to do things are found. (Reference the US Dept. of Labor Statistics via YouTube’s “Shift Happens”) Bluntly, the rate of change is mind numbing, and it has very little to do with software or technology directly, but more the adoption and utilization of technology through a shift in demographics and world economics.
Bringing it back down to Earth: The Federal Aquisition Lifecycle is no longer aligned with reality for software and systems implementation.
Example: during one “requirements” development phase for an RFP with one government client, we utilized the SCRUM Agile Development Methodology and Open Source development tools to iteratively build a proof-of-concept that demonstrated how the requirements would be implemented. This occurred over six, two-week iterations with releases of the software proof-of-concept at four, six, eight, ten and twelve weeks. Feedback was captured every two weeks during the twelve week “project”. Requirements were captured and validated in the team WIKI at every planning meeting (i.e. No Paper, no documents, just a list of well thought-out requirements). The requirements were turned into something called “User Stories” which in Software Engineering equates to lower level requirements that can be written as test cases. As user expectations changed, the requirements changed and the user stories changed. Every two weeks the requirements, user stories and changes were re-shuffled in priority so that the software still worked with each new release (again… every two weeks). At the end of the twelve-week “proof-of-concept” we were done. DONE DONE. There was no longer a need for an RFP, no longer a need for a three-to-six month lag in the bidding process. No vendor manipulation of the bidding process through their relationships with government executives. Just Plain DONE. We went through a separate Independent Validation and Verification (IV&V) because nobody believed we were really done.
If I contrast this “rouge” task order with how traditional life-cycles look inside the government you begin to see the problem: it is not in the interests of most federal contractors to deliver this way and it isn’t in the interests of legislators who approve the FAR to allow work to be done this way. Once the job is done done, then you have to find the next job instead of drawing out the lifecycle in what has become the middle-class welfare system. In reality, if you deliver this way, people beat a path to your door and there is no end of work because… people are smart and come up with new ways of doing things.
John has hit the nail on the head: you can talk about what you are going to do and create a requirements document, or you can just get it done and document along the way. My vote is for doing and not wasting OUR money talking.
ooo. This sounded really terse. Sorry. 🙂
May 28, 2009 at 2:40 pm #72726
I believe a core issue is an inability of the Government to define what success looks like for a program. I teach classes on performance based acquisition and there is little to no understanding of how to tell when your program is successful. Developing a sense of where the program is trying to go and how to measure when you are there is far more important than getting industry to help write the requirements. Who should know best what a successful program looks like? Trouble develops when the Government tries to write specs instead of define and measure outcomes. You then get tied up in all the knots of whose solution is it?
Taking the time up front to define some clear goals is important. If the Government can define clear goals and then solicits industry for fair measures then I believe there is a role there for collaboration. When success is defined, figuring out how we measure when we are there is a different exercise. You can also allow for various ways to get there and also the flexibility that John mentions.
May 28, 2009 at 2:48 pm #72724
After 30 years of acquisition experience in public and private sectors, in diverse technologies, and in hardware and services, here are my conclusions on the impediments to good requirements definition —
1. A general decline in general literacy in society. By this, I don’t mean simply the ability to string words together in a sentence with proper syntax, spelling, and punctuation. I mean the higher orders of literacy – the ability to fashion the nuances of language to convey a concept or detail effectively to the intended audience. This command of language is something that has been lost in electronic stimuli, lowered education expectations, and sheer laziness.
2. An excess of collaboration and review – the dark side of group think. Individual responsibility to describe a requirement has been diluted and undermined by an excess of participation and inclusion. Too many cooks have spoiled the broth. Cohesion and coherence has been lost in committee-generated products.
3. Fewer program managers and “SMEs” are around who have gained the expertise by walking the walk. The peer-to-peer relationship between program staffs and industry counterparts that once existed has diminished. As a result, the message in a requirements document isn’t as clear or persuasive as it once was.
May 28, 2009 at 2:50 pm #72722
“also better to fail on a small scale faster, than one big large failure at the end.”
Hmm, sounds like somebody’s been reading Dan Ward!
May 28, 2009 at 5:04 pm #72720
I heard a great presentation yesterday by Mr. Seaman from the DoD’s Business Transformation Agency, where they talked about their “IT-360” concept. Their goal is that within 180 days of having a requirement for a capability brought to them, they will have a contract awarded. Within 30 days of contract award (day 210 of process), a 25% capability is delivered by the vendor. Vendor proposes what that functionality will be. At day 360, you get Initial Operating Capability in the hands of the users. FOC follows sometime in the next 2-3 years but with deliveries being made every 180 days of increased functionality/capability. That gets users involved and using the systems early. For those used to Agile development concept, this isn’t new. But it’s news that a large DoD organization wants to adopt this as the standard.
Mr Seaman’s best line of the presentation – “In DoD, systems are like cows – once you name them, you can;t kill them”
May 31, 2009 at 4:36 am #72718
Good discussion and points…GREAT points on agile development – must have been a great group of programmers; project manager and government involvement. In that instance the government should have put together a set of requirements for agile development capabilities – not the requirements of the system itself. And procured on either a T&M or CP basis for the staff.
I think the larger problem than the requirements development is the evaluation process. The government is often unwilling to get specific on evaluation scores and end up lumping scores into big bands – like “ACCEPTABLE” or GREEN. Then it becomes a price shoot out. This way the government cannot be blamed for picking one company over the other – hey if you are both GREEN based on subjective reasoning there is one thing the government can be objective on – price.
So the government is forced to choose – a few requirements that leave big holes for low bids – or a lot of requirements which restrict flexibility in execution and creativity in solutioning.
As to the core question – would developing requirements with contractors in a wiki work? How would the government control user inputs – what if one company works harder on the requirements; is more vigilant and removes other recommended requirements that it knows are not to it’s advantage? How would the government keep up? Multiple IP addresses offering “independent” thoughts but actually a coordinated campaign…
I think a government-only wiki where requirements can be shared is a good idea (how many people develop requirements for Exchange server management every year). After that the government should find a way to post the requirements and invite comments from industry similar to the proposed rule process (regulations.gov). The comments should be submitted on-line; in a public forum; with only one submission per company with revisions allowed up to a closing date.
June 3, 2009 at 5:11 pm #72716
Writing requirements is a task the when preformed, reflects the background(s) and prejudices of the persons creating the content. That statement is not a statement of judgement it is a fact that people write what they know (if not is a novel). What is the state of Performance Based Contracting? The fundemental idea behind the appraoch is to create a TStatement of Outcomes and let the different vendors bring their corporate intellect to the table. Telling the vendors how to do the job seems to be in the DNA of people writing SOWs. It is a little like asking some for the time and telling then how to build the watch. I realize this is a bit of an over simplification, however it is not disingenuous. Maybe we should look at retooling the Requirements Development process. I think it is worth a discussion.
June 3, 2009 at 5:25 pm #72714
Mary, you bring up a good point. One of the methods the government uses that I always believe is a waste of time is when and agency issues a Request for Information. The agency is usually fishing for what solutions might be out there. Wouldn’t it be better to use a web 2.0 collaborative tool, allow vendors to comment to some general requirements for a period of time? This would allow vendors to not reveal secrets but have a meaningful discussion on is the govt. asking for is reasonable and ask questions in closer to real time versus the long process of an RFI. The govt. might find out sooner if what they are seeking is reasonable.
June 6, 2009 at 3:15 am #72712
Very interesting thread! Two paths I’d like to address – requirements generally, and agile development specifically
Going back to Mary’s original questions, I like the idea of a requirements wiki, but am skeptical of what industry’s participation might be (competitive advantage, tipping one’s hand, lurking, etc.). I am new-ish to gov’t (5 years last month!) and very new to federal IT program management (team member). Boy, have I seen some clunkers when it comes to poorly written PWS or SOW! Yikes, it’s expensive not to a) know what you want, and b) not have the savvy or interest in overseeing your contractors.
Let me also admit to a somewhat jaded view of Agile development (I admit, perhaps ill informed), but my observation of the execution of the agile concept is that both program offices (customers) and contractors HATE to document what they want, never mind documenting what they get in code and what changes are being made to it over time. That’s not even getting into whether developers with production access truly need that level of access 24×7, and who is minding the store while they are wandering the aisles. Okay, enough ranting on that!
What I continue to observe in my limited experience is that we tend to have poor business processes (paper based transactions being hand carried from office to office and regularly getting lost, for example) which both permits and, for those so-inclined, encourages exploitation of the lack of internal controls for a given program area. I’m all for speed and efficiency and being done, but how did you get there? Is it repeatable, documented, and able to withstand the scrutiny of IG or GAO or DCAA? Can someone unfamiliar follow your bread crumbs (or twitters today I guess) and arrive at the same place?
The other piece of this is the “soft side,” i.e. the people involved. Are they the best/brightest or transferred without their consent/consultation into an area they feel uninterested in or unfamiliar with? Are people creating good process, good strategy, innovative approaches getting enough traction, management support, encouragement from colleagues to do what’s right over what’s expedient?
Great to watch this conversation unfold – yay GovLoop!
June 6, 2009 at 1:15 pm #72710
I have to agree with Jim B. Most RFI’s are a waste of time that could be better spent getting RFPs out the door and evaluated with more “velocity”. More time should be invested writing a well crafted SOW that describes the needed service or product in terms of what the govt is trying to achieve (and any constraints). I certainly agree that there have been some horrendously bad SOWs created recently. I advocate for doing some experimenting with WIKIs to work on draft SOWs for new technologies that we would like to see used in govt. Open it up to interested parties (govt and industry) for their insights and additions. The COTR can evaluate the results of the collaborative effort and make final decisions on the SOW content. Another alternative would be to have the COTR write a draft and use the WIKI to collect questions/suggestions from the industry or other govt parties intended to improve or clarify parts of the SOW. Govt already does this when receiving comments to a draft RFP. We could just make the SOW refinement/question and comment process happen FASTER. A worthy goal in itself.
June 8, 2009 at 8:29 pm #72708
Great observations and comments. I believe that performance based contracting is still the best option in the long run. I suspect it could be argued that certain acquisitions do not lend themselves well to performance based contracting.
June 11, 2009 at 12:11 pm #72706
Kim, I believe your statement “Wikis might be appropriate to chronicle institutional memory for a design process, but 2 or 3 phased processes for structured communications and social collaboration might be better in for instance, a process that is built on alternative analysis” is key. I was speaking with someone last night who is with a company that is transitioning from being a small business to a large business. We were discussing the concept of collaborative, open requirements development and he pointed out that as a small business, resources are very limited requiring the company to make calculated choices about what work they bid on with those limited resources in contrast to larger companies who can bid on more work and spend much more money on each of those bids.
He applied this line of thinking to the concept of using an open development wikified platform for requirements development. He believed it would be hard for a small business to “keep up” with and respond to the ideas flying around in the cyberspace environment and felt again, that large businesses could have teams of people researching, generating ideas and responding to others’ ideas that would (or could) sway the requirements development process and potentially steer the resulting requirements document in the favor of those with most resources to throw at it. We both agreed the Long Tail would probably apply well in this situation which would mean we’d have to understand the underlying relationships of the factors that would impact the parity in the process.
I also believe there is great value in some of the traditional forms of communication we often use in conducting market research and developing requirements and that an online collaborative approach would not necessarily replace those other channels but could be used in addition to, for complementing and augmenting traditional processes while adding transparency. So your idea of “2 or 3 phased processes for structured communications” might be an approach to help level the playing field.
June 13, 2009 at 12:07 pm #72704
LeRoy Dutton, Jr.Participant
The words “collaborative” and “transparent” point directly toward a 2.0 From an industry point of few, it could provide an open method of with potential vendors – especially, when gathering information (market surveys) and feedback b4 a solicitation.
June 19, 2009 at 3:33 pm #72702
Here’s a link to the short article I wrote for FCW related to our group discussion – http://fcw.com/articles/2009/06/22/comment-davie-acquisition-2.0.aspx
Keep the dialogue going!
June 21, 2009 at 11:04 pm #72700
Kim Patrick KobzaParticipant
Mary, One of the thoughts that is fascinating about your comment is the likely tension between the centralized, institutional, vendors and the small business long tail vendors. Could the institutional vendors “overwhelm” the process?
The ultimate goal of any Gov2 strategy should in large part be to achieve the best result, and most options for the government and its stakeholders. Generally speaking gov 2 tools are viewed as a low cost way to enable the long tail – small businesses. Theoretically, they should be able to touch a greater subset of a limited population of opportunities. They should also be able to advance more possible solutions for government – the quality of ideas rising to the top. But that can only happen if all ideas start on an equal footing.
We might say that today large vendors that aggregate resources have advantages. And undoubtedly smaller vendors also have advantages in many others – especially where highly specialized knowledge is required. Gov 2 or in this case acquisition 2 could and maybe should shuffle the deck in ways that we can not anticipate.
Chances are good that application of 2.0 to acquisition much as in the private sector will generate emergent properties that we do not expect. And to complicate our decision making, it might take a while for us to be good at it, or to fully understand how to best leverage.
Chances are good that we aren’t going to get acquisition 2 exactly right, at inception. But the alternative is status quo and we have seen in private sector transformations that stasis is not a viable option.
June 24, 2009 at 2:48 pm #72698
Devin, Agile is not a panacea. Check this blog post out which reviews a very interesting article that convincingly demonstrates situations in which Agile actually stifles innovation:
“How Can Collaboration Systems and Social Media Complement Agile Project Management?”
Regarding requirements, I totally agree that we need to be able to respond to changing situations, and I’d be happy if statement of work would at least describe “who what when where when how” in enough quantitative and qualitative detail to enable a contractor other than an incumbent to understand situations that people are increasingly being asked to bid on on a fixed price basis!
Dennis D. McDonald, Ph.D.
Latest blog post:
“For Government Contracting, Defining “Inherently Governmental” is Inherently Difficult”
June 27, 2009 at 10:38 am #72696
Received some comments from a friend of mine and thought I’d share – I thought the e-Records management tool he mentions was interesting and probably necessary:
I think you could probably do some of the collaborative development of requirements now if you limited it to within an Agency (or in collaboration with a supporting Agency like GSA). Ideally, the CO should be integrated with the requirements development process from the beginning and should get feedback from all of the internal stakeholders on the requirement as defined. That sounds good, but is very difficult as you know. If the CO had some sort of e-Records management tool that they could rely on to share documents and see who makes the modifications that would be extremely helpful. (GAO has an electronic document management system that tracks changes, mods, etc. Corporately we use a platform called Privia (I don’t know who makes it) to do the same thing).
In some form it might make sense to capture the requirements development discussions in a manner that the source selection authority or board can use to help them in understanding the requirement.
Hope all is well.
June 30, 2009 at 2:15 pm #72694
More often than not, the government has used contractors to develop software. Given the iterative nature of the software development process and the interest in bringing work in-house with government employees, it might be time for us, the governmnt, to create software development centers. Wonder if our pay scales would match industry offerings?
June 30, 2009 at 2:22 pm #72692
Peter G. TuttleParticipant
It might be tough to match pay scales, but with today’s economy – who knows? One concern I have about bringing work in-house is how the gov’t would ensure that skills, knowledge & experience are routinely refreshed so gov’t workers can stay up with rapidly evolving technology, processes, etc. in the commercial marketplace. This is an interesting and important subject and I look forward to seeing other community member’s chime in.
June 30, 2009 at 5:33 pm #72690
I’m coming to this idea a bit late, but we need to figure out how to make crowdsourcing a viable option for Requirements Development. This is at some level what we are attempting to do with DefenseSolutions.gov. Vice letting the government spend oodles of time defining the requirement, to the extent practicable, open up the conversation by sharing the problem, and allow quick seed money to for people & organizations who come up with innovative ideas for solving it. This in effect becomes a crowd-sourced prototyping approach for the large projects, and a vastly cheaper solution for the smaller ones.
To do this Federal-wide, you need the equivalent of Other Transactions Authorities available to all agencies. Perhaps with a cap at 200K or something. This would allow for real up-front innovation, while still requiring larger purchases to go through a rigorous contracting process.
July 1, 2009 at 1:02 pm #72688
The Due Diligence sessions we use prior to solicitation release could be initiated earlier in the requirements development phase. Get the government porgram personnel a chance to be ‘out and about’ earlier. Another idea is to use the present climate to “in-source” (vs A-76) to build a cadre of government employees to do software development because development projects do take a long time and can easily exceed the 5 year limit of most contracts. It might also go a long way in eliminating the perception of the present Administration that we award too many limited competitive contracts.
March 14, 2011 at 9:49 pm #72686
I believe all the factors you listed–resource shortages, constrained timelines, the siloed structures in which such requirements are developed and managed, not enough collaboration between government and providers before programs/projects begin–all contribute to poor requirements. However, we find that the biggest factor is the absence of someone in a team (integrated or otherwise) who owns the requirements and is responsible for shepherding them throughout the program or project’s lifecycle. More often than not, requirements management and development is shared among program/project managers, the contract manager, the COTR, etc. There is therefore no one dedicated to working with the business process owner on ensuring those requirements continue to reflect the needs of the agency or bureau through changes in administration, budgets and technology. I’d like to share with you “The CIO’s 26th Point? Requirements Management and IT Management Reform”, which tackles the requirements issues within government. Please feel free to read and share with your contacts.
You must be logged in to reply to this topic.