700 retraceable reconstructable

600 bricolage

• the best criterion for rigour: replayability, retrospective reconstructability: even if it wouldn't have been obvious a priori how to answer this question, afterwards you can see how these conclusions were arrived at and can retrace the steps yourself.

• No method, not even RCTs, gives you a free pass to not do evaluative thinking. Because they're always embedded in the judgement of the evaluator that this is the right thing to do and I've checked all loopholes. In the same way our reasoning when doing bricolage is evaluative reasoning. The real-life application even of what seems like a monolithic tool like an RCT involves at different levels dozens of evaluative judgements about what to accept and what not to accept, whether this questionnaire or econometric measure is up to scratch, whether to trust this researcher, etc., whether this quantity of missing data is acceptable, etc.

• This appeal to evaluative reasoning is in some sense a transcendent one: you can never completely transfer responsibility to an algorithm. Because you always at least bear responsibility for using this particular algorithm in this particular way.

• At the same time algorithms can be incredibly helpful in breaking down our reasoning into transparent steps.

• Nachvollziehbarkeit also means it fits in your head

• Bricolage, rigour and Nachvollziehbarkeit in a partially evaluative summary of the Strategic Plans of 191 Red Cross / Red Crescent societies globally.

  Steve Powell, Mon, May 20, 2024

• But what I find weaker in this paper is the just the generic weakness of the constructivist response to classic Cambellian validity:

• The bomb that's just dropped is suddenly we do have ways thanks to generative AI of making qualitative judgements (about qualitative constructions, about people's meanings, people's beliefs and understandings) which are in some sense reproducible and intersubjective and reliable. The methods aren't essentially any different from before, but they can be done quickly, reproducibly and at scale and in a way which vastly reduces noise due to the individual researcher. (Our chosen AI is of course also in some sense biassed, but it's the same bias each time, so we can compare results across timepoints and groups in a way which was never possible before: we can claim our methods are reliable).

• As a footnote here are two members of the extended family of the concept of validity: