Staff Blog


What Is The Minimum Acceptable Value For An Attribute Agreement Analysis

  • 2021年4月16日
  • 未分類

Attribute analysis can be an excellent tool for detecting the causes of inaccuracies in a bug tracking system, but it must be used with great care, reflection and minimal complexity, should it ever be used. The best way to do this is to first monitor the database and then use the results of that audit to perform a targeted and optimized analysis of repeatability and reproducibility. Finally, and it is a source of additional complexity that is inherent in faulty database measurement systems, the number of code choices or locations can be heavy. Finding scenarios to study the reproducibility and reproducibility of any disease can be overwhelming. If the z.B database has 10 different error codes that could be assigned, the analyst should carefully select the scenarios to provide an appropriate representation of the different codes or locations that could be assigned. And realistically, a selection of 10 different categories for the type of error is at the bottom of the scale of what bug databases often allow. Analytically, this technique is a wonderful idea. But in practice, the technique can be difficult to execute judiciously. First, there is always the question of sample size.

For attribute data, relatively large samples are required to be able to calculate percentages with relatively low confidence intervals. If an expert looks at 50 different error scenarios – twice – and the match rate is 96 percent (48 votes vs. 50), the 95 percent confidence interval ranges from 86.29% to 99.51 percent. It is a fairly large margin of error, especially in terms of the challenge of choosing the scenarios, checking them in depth, making sure the value of the master is assigned, and then convincing the examiner to do the job – twice. If the number of scenarios is increased to 100, the 95 per cent confidence interval for a 96 per cent match rate will be reduced to a range of 90.1 to 98.9 per cent (Figure 2). An attribute analysis was developed to simultaneously assess the effects of repeatability and reproducibility on accuracy. It allows the analyst to review the responses of several reviewers if they look at multiple scenarios multiple times. It establishes statistics that assess the ability of evaluators to agree with themselves (repeatability), with each other (reproducibility) and with a master or correct value (overall accuracy) known for each characteristic – over and over again. However, a bug tracking system is not an ongoing payment. The assigned values are correct or not; There is no (or should not) grey area. If codes, locations and degrees of gravity are defined effectively, there is only one attribute for each of these categories for a particular error. Despite these difficulties, performing an attribute analysis on bug tracking systems is not a waste of time.

In fact, it is (or may be) an extremely informative, valuable and necessary exercise. The analysis of attributes should only be applied with caution and with a certain focus.