【算法检验】StepM和MCS
继上节SPA继续介绍StepM和MCS检验
bootstrap检验方法⼀、StepM检验
1、概念
SM类似于SPA，都具有同样的零假设。主要不同：SM可以识别⽐基准模型好的⼀系列模型，⽽不是只是验证某个模型更好。
Stepwise Multiple Testing is similar to the SPA and has the same null. The primary difference is that it identifies the set of models which are better than the benchmark, rather than just asking the basic question if any model is better.
2、例⼦
from arch.bootstrap import StepM
stepm = StepM(bm_losses, model_losses)
stepmpute()
#识别出优于基准模型的模型
print("Model indices:")
print([model.split(".")[1] for model in stepm.superior_models])
#求各模型的平均Loss
test2 = an(0)
better_models = pd.concat([an(0), an(0)],axis=1)
lumns = ["Same or worse", "Better"]
#得出better模型的index
better = better_models.index.isin(stepm.superior_models)
#得出worse模型的index
worse = np.logical_not(better)
#写为正确的值
better_models.loc[better, "Same or worse"] = np.nan
better_models.loc[worse, "Better"] = np.nan
#画图：绿⾊为更优的模型
fig = better_models.plot(style=["o", "s"], rot=270)
⼆、MCS检验
1、概念
The Model Confidence Set
MCS将⼀系列Loss作为输⼊，以发现类似的算法集。其主要输出为P值，其模型的P值⾼于Size。较⼩的P值代表模型能够被其他更好的模型拒绝。
The model confidence set takes a set of losses as its input and finds the set which are not statistically different from each other while controlling the familywise error rate. The primary output is a set of p-values, where models with a pvalue above the size are in the MCS. Small p-values indicate that the model is easily rejected from the set that includes the best.
2、例⼦
from arch.bootstrap import MCS
# 取出25个算法，500/20=25
losses = model_losses.iloc[:, ::20]
mcs = MCS(losses, size=0.10)
mcspute()
# 显⽰各模型的MSC P值
print("MCS P-values")
print(mcs.pvalues)
# Included: P>0.1
print("Included")
included = mcs.included
print([model.split(".")[1] for model in included])
# Excluded: P<=0.1
print("Excluded")
excluded = luded
print([model.split(".")[1] for model in excluded])
Included为更优秀的算法集，P越⼤，算法越好。