Recent test from Oregon State University estimated that more than 3,500 animal species are at risk of extinction due to factors such as habitat change, overexploitation of natural resources and climate change.
To better understand these changes and protect vulnerable wildlife, conservationists like MIT graduate student and Computer Science and Artificial Intelligence Laboratory (CSAIL) researcher Justin Kay are developing computer vision algorithms that closely monitor animal populations. Kay, a lab member in MIT's Department of Electrical Engineering and Computer Science, and CSAIL principal investigator Sara Beery, is currently working to track salmon in the Pacific Northwest, where they provide key nutrients to predators such as birds and bears while managing populations of prey such as worms.
But with all this volume of wildlife data, researchers have a lot of information to sort through and many AI models to choose from to analyze it all. Kay and his colleagues at CSAIL and the University of Massachusetts Amherst are developing artificial intelligence methods that greatly increase the efficiency of the data processing process, including a new approach called “consensus-based active model selection” (or “CODA”) that helps conservationists choose which artificial intelligence model to use. Their Work was named the top paper at the International Conference on Computer Vision (ICCV) in October.
This research was supported in part by the National Science Foundation, the Natural Sciences and Engineering Research Council of Canada, and the Abdul Latif Jameel Laboratory for Water and Food Systems (J-WAFS). Here, Kay discusses this project as well as other conservation efforts.
Q: In your article, you ask the question which artificial intelligence models will perform best on a specific data set. Given that there are as many as 1.9 million pre-trained models available in the HuggingFace Models repository alone, how does CODA help us overcome this challenge?
AND: Until recently, using artificial intelligence to analyze data usually meant training your own model. This requires significant effort to collect and describe a representative training dataset, as well as iterative training and validation of models. You also need a specific set of technical skills to run and modify AI training code. However, the way people interact with AI is changing – in particular, there are now millions of publicly available, pre-trained models that can perform a variety of predictive tasks very well. This potentially allows people to use AI to analyze data without having to develop their own model, simply downloading an existing model with the features they need. But this poses a new challenge: which model, out of the millions available, should you use to analyze your data?
Typically, answering the model selection question also requires spending a lot of time collecting and annotating a large dataset, albeit for the purpose of testing models rather than training them. This is especially true for real-world applications where user needs are specific, data distributions are unbalanced and constantly changing, and model performance may be inconsistent across samples. Our goal with CODA was to significantly reduce this effort. We do this by setting the data annotation process to “active”. Instead of requiring users to simultaneously annotate a large set of test data in bulk, with active model selection we make the process interactive, guiding users to annotate the most informative data points in their raw data. This is extremely effective and often requires users to annotate just 25 examples to identify the best model from a set of candidates.
We are very excited that CODA offers a new perspective on how to best leverage human effort in developing and deploying machine learning (ML) systems. As AI models become more common, our work highlights the value of focusing efforts on robust assessment processes rather than solely training.
Q: You used the CODA method to classify wildlife in photos. Why did it work so well and what role could such systems play in monitoring ecosystems in the future?
AND: One key insight was that when considering a set of candidate AI models, the consistency of all their predictions provides more information than the predictions of any single model. This can be seen as a kind of “wisdom of the crowd.” Combining the votes of all models gives you, on average, a decent understanding of what the labels of individual data points in the raw dataset should be. Our approach to CODA is based on estimating a “confusion matrix” for each AI model – given that the true label for some data point is class X, what is the probability that an individual model predicts class X, Y, or Z? This creates information dependencies between all candidate models, the categories you want to label, and the unlabeled points in the dataset.
Consider an example application where you are a wildlife ecologist who has just collected a dataset containing potentially hundreds of thousands of images from cameras deployed in the wild. You want to know what species are in these images. This is a time-consuming task that computer vision classifiers can help automate. You're trying to decide which species classification model to use on your data. If you have labeled 50 tiger images so far and a certain model performed well on those 50 images, you can be sure that it will perform well on the remaining (currently unlabeled) tiger images in the raw dataset. You also know that if this model predicts that an image contains a tiger, it will likely be correct, and therefore any model that predicts a different label for that image will be more likely to be wrong. All of these interdependencies can be used to construct probabilistic estimates of each model's confusion matrix, as well as the probability distribution of which model has the highest accuracy over the entire dataset. These design choices allow us to make more informed decisions about which data points to label, and are ultimately why CODA makes model selection much more efficiently than in previous work.
There are also many exciting opportunities to expand our work. We believe that there may be even better ways to construct informative model selection priorities based on domain knowledge – for example, if one model is already known to perform exceptionally well in some subset of classes or poorly in others. There are also opportunities to expand the platform to support more complex machine learning tasks and more sophisticated probabilistic performance models. We hope that our work will be an inspiration and a starting point for other researchers to constantly improve their knowledge.
Q: You work in Beerylab, led by Sara Beery, where researchers combine the pattern recognition capabilities of machine learning algorithms with computer vision technology to monitor wildlife. Outside of CODA, how else does your team track and analyze the natural world?
AND: The lab is a really exciting place to work and there are new projects emerging all the time. We carry out projects monitoring coral reefs using drones, re-identifying individual elephants over time, and combining multi-modal Earth observation data from satellites and in-situ cameras, to name just a few. Broadly speaking, we are looking at emerging biodiversity monitoring technologies and trying to understand where the data analysis bottlenecks are, and developing new computer vision and machine learning approaches that solve these problems in a broadly applicable way. This is an exciting way of approaching problems that, in some sense, focuses on the “meta-questions” that underlie the specific data challenges we face.
Examples of this work include computer vision algorithms I have been working on that count migrating salmon in underwater sonar video. We are often faced with changing data distributions, even as we try to construct training data sets that are as diverse as possible. When we install a new camera, we always encounter something new, which usually degrades the performance of computer vision algorithms. This is one example of a general machine learning problem called domain adaptation, but when we tried to apply existing domain adaptation algorithms to our fisheries data, we realized that there were serious limitations in how existing algorithms were trained and evaluated. We managed to develop a new domain adaptation framework, published earlier this year Machine Learning Research Transactionswhich addressed these limitations and led to advances in fish counting and even in the analysis of autonomous vehicles and spacecraft.
One area of work that I am particularly excited about is understanding how to better develop and analyze the performance of predictive machine learning algorithms in the context of what they are actually used for. Typically, the output of some computer vision algorithm—say, the bounding boxes of animals in images—isn't actually what people care about, but rather a means to an end to answering a larger problem—say, what species live here, and how does that change over time? We have been working on methods for analyzing predictive performance in this context, and with this in mind, we have reconsidered the ways in which we use human knowledge in machine learning systems. One example was CODA, where we showed that we could actually consider the machine learning models themselves as constants and build a statistical framework to understand their performance very effectively. We have recently been working on similar integrated analyzes combining ML predictions with multi-stage prediction pipelines as well as ecological statistical models.
The natural world is changing at an unprecedented pace and scale, and the ability to quickly move from scientific hypotheses or management questions to data-driven answers is more important than ever to protect ecosystems and the communities that depend on them. Advances in artificial intelligence can play an important role, but we need to think critically about how we design, train and evaluate algorithms in the context of these very real challenges.
