Read about the necessary but terrifying process of having our MERCURY model replicated. How robust were our previously published results? Hilde Kanters’ excellent work! With Iza Romanowska and myself.
Read the open access paper here https://www.sciencedirect.com/science/article/pii/S2352409X21001863
Jeroen Poblome and I published the MERCURY model in 2016 https://doi.org/10.15184/aqy.2016.35… We performed a selected set of experiments that allowed us to explore the model’s behaviour. But how robust were the results that we published there?
Our original experiments suggested that weak market integration (low availability of reliable non-local information), equal production capacities of pottery manufacturers, and equal demands at settlements throughout the roman world were unlikely to explain tableware distributions. We came to these conclusions by performing 34 experiments where we changed the relevant parameters of the MERCURY model that represented key explanatory factors. These experiment settings are shown in detail in our 2016 paper in JASSS: http://dx.doi.org/10.18564/jasss.2953
34 experiments is pretty good for an archaeological model, I tell myself. And certainly they suggest how the model behaves. But they do not reflect the full range of theoretically possible scenarios: we did not originally explore the full possible parameter space.
In comes the fantastic Hilde Kanters, who independently performed a replication of the MERCURY model in Repast Symphony (the original was coded in Netlogo) using only our publications. Her MSc thesis at Leiden University. Imagine my terror/excitement when I heard.
The replication study came to substantively the same conclusions as we did *massive sigh of relief* and made some very critical but constructive recommendations *massive collegial handshake*
This replication study is available open access https://studenttheses.universiteitleiden.nl/handle/1887/68248
The replication was made possible because not only did we publish the technical side of the model in detail in JASSS but mainly because we made the model code openly available on Comses OpenABM https://comses.net/codebases/4347/releases/1.1.0/
We were so impressed with Hilde Kanters’ work and delighted when the opportunity arose for her to do an internship in project MERCURY when I was at UBICS in Barcelona (such a great place to be as a complexity scientist). And Iza Romanowska luckily co-supervised this internship, which really set it up for excellent outputs. But what to do? I had loads of fun ideas to expand MERCURY: transport routes, equilibrium model, pots in spaaaaaaaaaice.
But in the end we knew we needed to do the responsible thing: check how robust the previously published results of MERCURY were by exploring the parameter space. We did a sensitivity analysis! See the results in our new paper in JASR. We now understand the model so much better, and I can be very confident of two key conclusions: the explanatory power of limited availability of reliable non-local information, and of strong differences in production capacity. But crucially, the sensitivity analysis also revealed I should be more cautious about the explanatory power of differences in demand throughout the Roman world: this was a new unexpected result, and will inform how I develop MERCURY in the future for sure.
Another thing that became painfully clear when working on this paper was the sad fact that ARCHAEOLOGISTS DON’T DO SENSITIVITY ANALYSES… But hey, we can help you on your way. We published a script for performing sensitivity analyses of ABM that can be reused by anyone, hurray! https://zenodo.org/record/4741208
Read more in the paper: https://www.sciencedirect.com/science/article/pii/S2352409X21001863
Abstract
Computational modelling is increasingly gaining attention in archaeology and related disciplines. With the number of new models growing it is often difficult to evaluate their significance and the generality of the results. This is partially due to the narrow reporting of the model’s results, which are often limited to those directly relevant to the research question posed in the first place. Although this is not an issue per se, models, if explored exhaustively, can provide a much wider perspective on the studied system. Sensitivity analysis is a widely recognised model exploration method for assessing the importance of different parameters on the model’s behaviour. Such systematic exploration helps in unravelling the dynamics that drive the model and enable researchers to establish how robust the presented results are. Here we present a sensitivity analysis of MERCURY, a previously published archaeological agent-based model. The results show that two out of three of the original conclusions drawn on the basis of selective experiment design stand up to scrutiny. By describing in detail and providing a reusable script with detailed description of all steps of the sensitivity analysis we hope to promote this important model exploration technique among modellers and the wider archaeological audience.
Archaeological Networks 