The future of the government forges

really great dissection of "forges" a must read

Written by Gunnar Hellekson

The future of the government forges

http://onepeople.org/node/2159
The GSA is currently planning forge.gov, which is widely assumed to be based on forge.mil, the much-discussed collaboration platform from the Defense
Information Systems Agency, or DISA. forge.mil is a pretty incredible
idea: a single destination for testing, certification, and software
development in the Defense Department.

It sounds obvious, but the idea remains revolutionary. For the first time, there would be a single repository for source code that could be shared between the hundreds of agencies, commands, and programs in DOD.
Developers would be able to share their work in a familiar, web-based
environment. A previous version of forge.mil was pulled for unknown
reasons, but the current iteration is based on the TeamForge product from
CollabNet. If you’ve used SourceForge, you get the idea. The DOD is the
largest consumer, and one of the largest developers of software in the
world. Much of this software is redundant, locked up by vendors and
integrators, can’t work with other software, and nobody remembers how to
maintain it. There’s no doubt forge.mil was long overdue.

It’s dangerous, though,  to assume that forge.mil is a useful template for forge.gov. I think forge.mil could lead forge.gov down the same road as core.gov and other failed attempts to encourage source code
reuse in government. To understand why forge.mil can be useful and
simultaneously poisonous to forge.gov, you have to first understand how
the DOD does software.

COTS vs. GOTS

Before, say, the mid-1990s, much of the DOD’s software was owned by the government. GOTS, or “Government Off-the-Shelf” as it’s now called, was built and maintained by the DOD and its contractors. This was
appropriate for some military-specific systems, but the strategy
outlived its usefulness when the government could no longer keep up with
commercial enterprises. For many pieces of common software, like
operating systems, spreadsheets and web browsers, the open market
produced more innovative and higher-quality products. So down came the
order: use commercial software. COTS (“Commercial Off-the-Shelf”) was
ascendant.

Boeing, Lockheed Martin, General Dynamics, and the other integrators no longer write as much software from scratch. Instead, their business model has changed. They now glue commercial software together, and wrote
code to fill in the gaps. Their business model, however, remains the
same. Instead of seeking rent on the proprietary software they’ve
written on behalf of the government, they seek rent on the integration
work they’ve done.

This is a lucrative business: the latest UAV is composed of hundreds of software systems, some commercially available, some written by the integrator. If I’m the integrator, I’m the only one who knows how all
the pieces go together and I can charge a handsome sum to anyone who’d
like to field their technology on my platform. Think of Apple’s
locked-up App Store, but flying through the air with a missile strapped
to its belly.

There’s nothing nefarious about this, of course. Integrators are doing what the market commands, and controlling access to platforms is an perfectly legitimate business model. It discourages reuse, though,
which means that it’s difficult for the DOD to effectively use the
software it purchased or developed. You can read more about this strange
market and its consequences in the excellent “Losing the
Softwar(e)” by my friend and fellow Open Source for America
member, John Scott.

Openness as a Desperate Act

So Rob Vietmeyer of DISA decided to borrow from the principles of the open source community and Internet governance. A more open and transparent
development process at DISA could remove barriers to reuse, encourage
collaboration, and discourage proprietary or closed systems. COTS
software is still king, but where the government needs to control its
own integration, set its own standards, and exercise stewardship over
its own infrastructure, it can still develop its own GOTS solutions
— this time, in an open, collaborative manner. Where GOTS was once
insular, slow-moving, and highly proprietary, it can now be produced at
lower cost and with lower barriers to entry for new innovations. Because
this “Open GOTS” is built using familiar open source methods, the
projects have a fighting chance of working together.

The Walled Garden and the Moral Hazard

So far, the DOD software problem sounds similar to the government software problem. Don’t misunderstand forge.mil, though. It’s not your typical open source development environment. If you talk with the
forge.mil project team, like Guy Martin of CollabNet, he’s quick to correct you
if you compare forge.mil to SourceForge. forge.mil is something very
different. With all the challenges of this profoundly broken market,
forge.mil had to make some serious sacrifices.

The first thing you’ll notice about forge.mil is that you can’t get to it. Access to forge.mil is severely restricted. To get access, you must have an official DOD Common Access Card (CAC) or have a DOD
employee sponsor you for an “ECA Certificate”. Dave Wheeler of the Institute for
Defense Analysis describes it as “gated development.” On the mil-oss mailing list, this is referred
to as the “CAC Wall”.

The CAC Wall has some unintended consequences, and raises some very difficult questions. First, it prevents many well-meaning developers who don’t work for the Defense Industrial Base from helping the projects
inside. There are literally tens of thousands of developers behind that
wall, and forge.mil keeps that group partitioned from both the public
and government employees without CAC or ECA credentials. The bargain is
that by keeping these undesirables out, they carve out a sense of safety
so skittish program managers are more likely to host their code. The
cost of that sense of safety is a much smaller audience than they could
muster hosting on more public platforms.

The CAC Wall also creates a dangerous an incentive to split communities. What happens if someone in the DOD wants to hack on a piece of open source code? They’ll host the hack at forge.mil, and the public
could never see those hacks again. If I’ve licensed my project under an
open source license, it’s because I want others to contribute. If that
code disappears behind the CAC Wall, I’m cut off from tens of thousands
of DOD developers. This “forge.mil fork” scenario is serious business.
It’s a scenario where everyone loses.

Finally, the CAC Wall may create a moral hazard for the developers who live behind it. In the open source community, folks are very careful about what code they commit, and since they’re never really sure who’s a
friend and who’s a foe. In a community where everyone’s “trusted,”
developers can become complacent, making them more vulnerable to poorly
written or hostile code.

Despite these real concerns, the CAC Wall still makes sense for forge.mil. DISA has some very legitimate security concerns about the code that’s being developed. They’ve weighed that risk against the
advantages of public scrutiny, cooperation with external projects, and
they’ve made the informed decision to keep the code behind the “CAC
wall”,  and to their credit they encourage forge.mil developers to
contribute patches upstream wherever possible instead of hosting on
forge.mil.

forge.mil as Role Model

So we have a kind of Judgement of Solomon in the forge.mil platform. Because of its very unique market dynamics, the DOD needs to take advantage of open source projects, the open source development model,
encourage collaboration, and reduce its reliance on proprietary
platforms. But for that to happen, DISA had to put all the work behind
the CAC wall.

forge.mil has attracted the attention of other agencies. How could it not? The DOD has the worst-case scenario: the gravest problem, the most complex market conditions, and the most dire consequences. So other
CIOs take notice and the press on this forge.mil experiment has been
relentless.

This brings brings us to the news that the GSA’s Dave McClure is planning forge.gov, a civilian counterpart to forge.mil. This is exciting.
This is also terrifying, because they seem to be following the forge.mil
model by restricting access to only US citizens. Guy
Martin says:

I realize that putting up a barrier to entry in the form of positive identification of US citizenship and a vetting process will irk some who believe that everything should be free and open…

Consider me irked. I want to be perfectly clear about this: the compromises made in forge.mil are dangerous for forge.gov. If forge.gov were to follow the forge.mil “CAC Wall” approach, it will permanently
damage the “Open GOTS” movement.

forge.gov cannot be forge.mil

forge.mil may be instructive and inspiring, but it’s a corner-case and fraught with compromises that have diminished its utility. In the case of forge.gov, it would be hosting unclassified code for civilian
agencies. There’s no need to create a “trusted” environment. There’s no
need to verify the citizenship or security clearance of its
participants. The standard open source mechanisms are more than
sufficient: only project leaders can commit code to the repository, a
semiformal review procedure for patches, and so forth. In any case, I’m
struggling to imagine why the repository would be better secured by
allowed access to 300 million people. Let’s agree that making
US citizenship a prerequisite is counterproductive, unworkable,
unnecessary and most important: it’s un-American.

The entire country of Jordan has adopted the VA’s VistA software for their national healthcare system. Countless overseas researchers collaborate with their US counterparts through open source projects.
When we share our source code with the world, it improves the quality of
the software and is, in fact, a uniquely practical kind of diplomacy.

Since we don’t need to control access to the projects as we do in forge.mil, I have to wonder why we would need a prescribed set of tools for hosting each project. TeamForge is a fine piece of software, but
there are literally dozens of viable alternatives. Developers are very
picky about their tools, and extremely picky about the version control
systems that are at the heart of these code repositories. If
we presumptuously select tools on their behalf, we create an unnecessary
barrier to entry.

The ideal forge.gov is two forge.govs.

With this in mind, we should break the forge.gov project in two.

First, forge.gov is useful as a catalog of open source projects that are used and created by the civilian government, many of which happily reside on agency websites or public
repositories already. Forge.gov could be the way that projects and
developers easily find each other. The Freshmeat or Ohloh of government, if you like.
Let’s call this the forge.gov Catalog. The forge.gov Catalog would be as
inclusive as possible, tracking the progress of every Open GOTS project
we can find.

There are still projects that need an infrastructure, of course, and forge.gov could provide that. The forge.gov Repository would be provided by the GSA as a service to agencies, and provide a complete development
environment. Here, I’m thinking of github or SourceForge.

I think by splitting these roles, and avoiding the dreaded CAC Wall, we can include the largest possible group of contributors and take best advantage of the excellent open source work that’s already underway.

Am I missing something here? Is there a national security concern that I’m overlooking? Is there an advantage to a homogeneous set of developer tools that I don’t understand? Let me know.