Evaluating Software Product Quality Through Systematic Audits and Conformity Analysis

The evaluation of software product quality is challenging, as traditional methods focus on discrete views that emphasize only parts of the software lifecycle, and thus provide a subset of the inferences relevant for the system as a whole. We describe a work-in-progress for a theoretical framework for software product quality evaluation that would be based on systematic audits and analysis of evidence, support graded assessment of conformity, and combine features that would allow the evaluation as a whole to be both holistic and repeatable. We describe our focus on Software-as-a-Service (SaaS) environments where it is common for the software provider to adopt a continuous deployment model and the customer to share responsibilities for software management. This is well-suited to assess development artifacts as well as operational controls. We explore how audits could be viewed as a quality evaluation technique that consolidates metrics-based approaches used in isolation, as well as enables evidence-based assessments. We present the conceptual basis of the framework and invite discussion of its application in various environments.