Earlier this week, while helping James prepare some words to include in this week’s EclipseZone newsletter on the Usage Data Collector (UDC), a thought occurred to me (which I included in the text): the information gathered by the UDC should help in testing.
Think about the magnitude of the testing problem that faces Eclipse projects. Consider just Ganymede. A user/organization is assembling a development environment will choose to either include, or not include each of these projects. That means, that there is an upper limit of 224 = 16,777,216 possible combinations of projects. This assumes that each user is going to take all the available bits of each project; if you try to factor in that each project produces some number bundles that can either be included or not included, then this number gets way bigger.
Now, with the assumption that each user/organization is going to take all the bits of each project, the number is actually a bit lower. To have a development environment, you need to have the Eclipse top-level project’s code, which drops the upper limit to 223 = 8,388,608 possible combinations of projects.
If each project has exactly one automated unit test that takes 1 second to run (which, if you consider the time required to startup the test environment is probably a little low), running the tests on all of the combinations will take, well… a heck of a lot of time (2,330 days). And that completely disregards any time required to actually configure all those different combinations. In fairness, the upper limit is probably (though not necessarily) a little high. Let’s say that the actual number of combination of projects in the wild is really two orders of magnitude lower than that upper limit (which brings us down to a scant 23 days of automated unit testing).
The point is that rigorously testing all possible combinations of Eclipse projects is—while not impossible—certainly impractical.
One of the handy things that the UDC captures is combinations of software that actually exist in the wild. We can tell from the data, for example, that certain features are actually used in combination. We will be able to tell, for example, how many people are actually using RAP in combination with BIRT. Or Subversive with Mylyn. Or Subversive and CVS in conjunction with TPTP and Buckminster. I have this vision in my head of a graph showing all the different bundles clustered into groups where the closeness between bundles indicates how commonly they are seen together in the wild.
From this, I expect that we’ll end up with a relatively small (i.e. much smaller than the upper limit) set of combinations of projects and bundles that are really used in the wild. From there, we can set priorities for testing, usability, interoperability, integration, etc.
What do we do with this information? At a minimum, we (“we” the Eclipse community, not necessarily “we” the Eclipse Foundation) might consider using the Eclipse Packaging Project (EPP) to build these most commonly found combinations to make them available for testing. It has been suggested that the Eclipse Foundation should invest some resources into testing; maybe having the packages based on real combinations in the wild is a step toward that, or maybe even something else entirely. Frankly, I think it’d be cool if we had a Testing project. At least we’ll know what to test.
I’ve added a new report that shows (1) how many people are participating each month, (2) how many new people are contributing data for the first time each month, and (3) how many events are uploaded each month. The report is updated weekly.
