My PhD
Title:
Incisive Tagging: Humans-in-the-Loop in Selection and Labelling of Remote Sensing Data Sets
External Stakeholder:
Ordnance Survey
The Research:
This project will be grounded in investigating the interplay of human creativity, intelligence and fulfilment with efficient ML tools. The work will begin and iterate around deep and intensive understandings of the labellers’ perspectives of the task; leading to prototypes and evaluations. These prototypes might involve novel gamification visualisation techniques; or even the use of multiple modalities - e.g. from simple gestures to emotional state recognition - to provide input to ML tools. The interaction design of the labelling tool will inform and be informed by algorithmic innovations within the ML tool. For instance:
One approach to making the task more efficient, immersive and less onerous would be to make spotting of pivotal examples easier. So, for example, we can present samples clustered on similarity as groups of thumbnails, allowing the labeller to spot outliers faster. Another strategy might be to use active labelling - i.e. given a small labelled data set can we present larger sets to users and gain their feedback to (a) label large amounts of data quickly and (b) resultingly make the labelling task less onerous.
We might also consider how to improve sample efficiency – that is, reducing the number of 25 samples required without reducing the efficacy of the approach. Some theoretical models on sample complexity have been investigated. Monte Carlo techniques would be a suggested research direction. For example, Importance Sampling has long been studied in Path Tracing and more advanced techniques such as Hamiltonian Monte Carlo or Gradient Domain demonstrate orders of magnitude performance gains through a great reduction in required samples. Ensembles are used to increase robustness and stability which lend well to importance sampling. We could also examine the literature on robust statistics and M-estimators as methods for drawing samples. In these sorts of investigation, we will draw on the labelers’ experience and insights to supplement any quantitative or theoretical evaluations of the power or limitations of the proposed approaches.
The human-centered improvements discussed above could also drive machine performance improvement. Models suffer from needing long training times. There is a potential that the current problem size can be compressed so it just fits into GPU memory to improve cache coherency during training. A suggested work package could build on the above (sample efficiency and improved clustering algorithms) to reduce the training set down to a few hundred thousand images that could fit into GPU memory resulting in an order of magnitude speed-up.