We collect articles from a variety of major online newspapers. We not only scrape international news sources, but also include major (by circulation) local newspapers within countries.
After we have collected the articles, they are run through a neural network that determines whether each individual piece of data relates to civic space changes. The neural network then codes data to a variety of civic space event types.
We then use all of our civic space event data and a 'Principal Component Analysis' (PCA) to generate a monthly civic space index. Lower scores indicate an increase in openness of civic space and a higher score indicates a decrease.
We combine the civic space event data with high-frequency economic data and then use regression analysis to forecast changes in civic space 1, 3 and 6 months into the future. More specifically, we use Lasso Regression in order to make the models more parsimonious and identify the key predictors of civic space.
Finally, we rely on a survey of local, country-based experts to ensure that we are accurately capturing news and events on the ground. Experts provide information on electronic sources they are reading, as well as key, recent events bearing on civic space. In return, we provide local experts with our forecasts as a tool they can use in their own work with civil society.