Write a PREreview

Comparative evaluation of intrusion detection configurations is almost universally reported as a single aggregate score obtained on one dataset. Such a protocol cannot separate the contribution of individual design decisions, cannot detect interactions between them, and provides no estimate of its own measurement error, which is one reason why laboratory results transfer poorly into operation. This paper sets out an experimental protocol that removes these defects, based on a full factorial design with replication. Three factors are varied at two levels each — detector type, threat intelligence enrichment and traffic conditions — in a 2³ layout with five replicates, giving forty runs; the complete treatment is specified, comprising the Cochran test for homogeneity of variances, the Student test for coefficient significance, the Fisher tests for model significance and lack of fit, the Shapiro — Wilk test for residual normality, and the variance inflation factor for multicollinearity. The protocol is validated under controlled conditions: the response is generated from a process model whose coefficients are known in advance, so that the recovered estimates can be compared against ground truth. The procedure reproduced all four generating coefficients within their confidence intervals (4.49 against 4.5; 1.59 against 1.5; −2.97 against −3.0; 0.96 against 1.0), yielded an adequate model (coefficient of determination 0.982; Fisher criterion 247.8 against the tabulated 2.31; lack-of-fit test 0.623 against 2.90), and correctly rejected the three effects that were absent by construction. The protocol detected an interaction of 1.92 percentage points, more than three times the error of the accompanying trend forecast, which indicates the resolution attainable at this design size. Applying the same protocol to traffic captured in an operating network is the subject of further work.