Evidence-based principles: notes.


Note1.

The usual caveat: correlation is not cause. The purpose of this analysis is to find objective correlations between properties and costs and it's true, such a correlation does not necessarily mean that reducing those properties also reduces costs. But having no objective correlations is not cause, either, and this is the situation we find ourselves in regarding most principles we use today. Consider this post as an attempt to examine our beliefs rather than to definitively justify them.

Note2.

The actual measurements made were as follows. If measuring size, then we measured the size of a class or package over all the times it was updated over multiple releases. Let's say it was updated five times. Then the two numbers correlated for this class or method were the number of times updated - 5 - against the average size over all 5 updates.

Also, it was noticed that some Java bytecode changes even when the source code does not (and even when the compiler versions do not change). As there was no way around this fault, this introduces some error into the analysis.

As usual, the caveat must be made that the only way to ensure that a class or package has been updated between project releases is to manually check it, and this was NOT done in this case. Instead, the project releases were sieved through a code analyzer which checked automatically for changes by comparing the before and after bytecode of each method. This will not catch all method changes, and could flag unchanged methods as changed. Hence the results of this experiment are not definitive.

Note3.

The programs analyzed were:

Only core jar files were analyzed where programs were very large.

The analysis requires that successive release use the same Java compiler version, as whether a method had changed is identified by whether the its bytecode changed. Hence Hadoop releases, for example, are split in two around a compiler update.

All data available on request.

Note4.

The properties investigated were:

  1. Size - size (in bytecode) of the given method.
  2. Dependencies from - number of dependencies leading from an method.
  3. Conditional count - number of conditionals in the bytecode, which roughly corresponds to conditionals in the source code, for example if-statements and loop boundary checks.
  4. Impact set - size of the impact set of a given method, that is, the complete set of all other methods that the given method depends on, either directly or transitively.
  5. Middle-man - shows whether an method is a middle-man, that is, it shows an method that could potentially be removed and instead have the parent do all the work that it did.
  6. Complectation - number of complected methods of multiple transitive dependencies. If a transitive dependency has multiple dependencies on another transitive dependency then it may be possible to access one method of the target transitive dependency through multiple paths. This sometimes (but not at all always) suggests an unnecessary duplication of method invocation, artificially raising the impact set of the system and thus exposing the system to unnecessary potential ripple effects.
  7. Transitive dependencies - number of all the transitive dependencies involving an method.
  8. Potential coupling - the absolute potential coupling of this method. For example, the absolute potential coupling of function A is the number of other functions that A could depend on, i.e., that are accessible from A.
  9. Amplification - the amplification generated by this method. This is essentially a measure of how the number of transitive dependencies in which this method is involved is increased as a combinatorial effect of dependencies on and from this method.
  10. Circular dependencies - number of circular dependencies between methods. For example, if function a() calls function b() and function b() calls function c() and function c() calls function a(), then a(), b() and c() form a circular dependency
  11. Dependencies on - number of dependencies on an method.
  12. Impacted set - This analysis shows the impacted set of a given method, that is, the complete set of all other methods that depend on the given method, either directly or transitively.)
  13. Duplication - number of common sequences of method invocations greater than a minimum value (currently 2). For example, if function a() calls functions x(), y() and z(), and function b() calls functions x(), y() and z(), then both a() and b() will be shown as sharing common calling sequences x(), y() and z().

Note5.

The Spearman correleation coefficients on class-level are:

Program S. D. f. C. c. T. d. I. s. A. p. c. A. C. d. Impd. S. C. M. D. D. o.
JUnit 0.38 0.13 0.35 0.2 -0.04 0.47 0.15 -0.1 0.13 -0.13 -0.06 0.14 0.22
ActiveMQ 0.25 0.23 0.22 0.19 0.04 0.23 0.24 0.17 0.18 -0.12 -0.13 0.15 0.18
Camel_1 0.3 0.22 0.25 0.22 0.07 0.35 0.21 0.11 0.17 -0.08 -0.11 0.11 0.2
Camel_2 0.25 0.16 0.22 0.15 0.06 0.29 0.19 0.11 0.14 -0.11 -0.14 0.1 0.15
FitNesse2 0.31 0.17 0.23 0.23 0.04 0.39 0.24 0.08 0.19 -0.04 -0.03 0.07 0.22
Hadoop_1 0.42 0.37 0.35 0.27 0.11 0.56 0.37 0.19 0.22 -0.21 -0.27 0.22 0.33
Hadoop_2 0.27 0.2 0.27 0.21 -0.06 0.45 0.27 0.15 0.29 -0.18 -0.24 0.16 0.37
Log4j_2 0.46 0.42 0.44 0.38 0.28 0.48 0.21 0.14 0.01 0.06 0.02 0.31 0.05
Lucene_1 0.33 0.21 0.3 0.29 0.06 0.42 0.23 0.17 0.18 -0.09 -0.05 0.2 0.17
Lucene_2 0.18 -0.02 0.1 0.07 -0.08 0.13 0.14 0.12 0.08 -0.23 -0.19 -0.02 0.14
Maven 0.25 0.1 0.24 0.12 0.04 0.36 -0.06 0.04 -0.04 0.05 0.06 0.12 -0.02
Struts 0.25 0.21 0.13 0.14 0.12 0.31 0.11 -0.01 -0.02 -0.04 -0.04 0.07 0.01
Zookeeper 0.47 0.34 0.43 0.39 0.12 0.55 0.36 0.3 0.29 -0.16 -0.14 0.24 0.31
Coyote 0.27 0.19 0.23 0.27 0.07 0.37 0.23 0.22 0.16 -0.09 -0.1 0.25 0.24
Derby 0.4 0.27 0.36 0.24 0.11 0.37 0.19 0.18 0.13 -0.11 -0.11 0.23 0.16
Avg. 0.32 0.21 0.27 0.23 0.06 0.38 0.2 0.12 0.14 -0.1 -0.1 0.16 0.18

Note6.

The Spearman correleation coefficients on package-level are:

Program S. D. f. C. c. T. d. I. s. A. p. c. A. C. d. Impd. S. C. M. D. D. o.
JUnit 0.3 0.3 0.24 0.33 -0.15 0.3 0.26 0.3 0.32 0.01 0 0.18 0.22
ActiveMQ 0.21 0.2 0.2 0.18 0.04 0.2 0.24 0.16 0.17 -0.1 -0.1 0.1 0.2
Camel_1 0.55 0.38 0.48 0.51 -0.06 0.61 0.35 0.35 0.47 -0.04 -0.04 0.52 0.42
Camel_2 0.39 0.26 0.38 0.32 0.02 0.44 0.29 0.19 0.35 0.04 0.03 0.36 0.32
FitNesse2 0.41 0.26 0.36 0.19 0.09 0.4 0.28 0.18 0.16 -0.09 -0.1 0.29 0.08
Hadoop_1 0.54 0.74 0.55 0.27 0.49 0.65 0.25 0.18 0.51 0.36 0.36 0.34 0.04
Hadoop_2 0.29 0.35 0.27 0.35 0.24 0.21 0.29 0.33 0.44 0.17 0.16 0.06 -0.03
Log4j_2 0.66 0.41 0.66 0.45 -0.01 0.63 0.14 0.54 0.12 -0.06 -0.08 0.54 0.15
Lucene_1 0.36 0.21 0.39 0.32 -0.35 0.39 0.32 0.24 0.31 -0.02 -0.03 0.33 0.31
Lucene_2 0.42 0.07 0.4 0.26 -0.12 0.35 0.33 0.25 0.35 -0.24 -0.23 0.23 0.32
Maven 0.42 0.36 0.46 0.45 0.21 0.45 0.4 0.2 0.42 0.03 0.03 0.31 0.44
Struts 0.29 0.35 0.2 0.16 -0.07 0.27 0.11 0.03 0.04 0.15 0.14 0.3 0.06
Zookeeper 0.32 0.25 0.32 0.15 0.26 0.36 0.26 0.1 0.1 0.04 0.04 0.28 0.14
Coyote 0.46 0.24 0.46 0.43 0.15 0.52 0.19 0 0.09 -0.04 -0.05 0.41 0.11
Derby 0.5 0.44 0.45 0.29 0.04 0.52 0.28 0.21 0.16 -0.17 -0.19 0.42 0.13
Avg. 0.41 0.32 0.39 0.31 0.05 0.42 0.27 0.22 0.27 0 -0 0.31 0.19