Dataset: An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models

Description

The open dataset, software, and other files accompanying the manuscript "An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models," submitted for publication to Integrated Materials and Manufacturing Innovations.Machine learning and autonomy are increasingly prevalent in materials science, but existing models are often trained or tuned using idealized data as absolute ground truths. In actual materials science, "ground truth" is often a matter of interpretation and is more readily determined by consensus. Here we present the data, software, and other files for a study using as-obtained diffraction data as a test case for evaluating the performance of machine learning models in the presence of differing expert opinions. We demonstrate that experts with similar backgrounds can disagree greatly even for something as intuitive as using diffraction to identify the start and end of a phase transformation. We then use a logarithmic likelihood method to evaluate the performance of machine learning models in relation to the consensus expert labels and their variance. We further illustrate this method's efficacy in ranking a number of state-of-the-art phase mapping algorithms. We propose a materials data challenge centered around the problem of evaluating models based on consensus with uncertainty. The data, labels, and code used in this study are all available online at data.gov, and the interested reader is encouraged to replicate and improve the existing models or to propose alternative methods for evaluating algorithmic performance.

Resources

Name Format Description Link
47 https://data.nist.gov/od/ds/mds2-2301/VO2%20-%20Nb2O3%20Composition%20and%20temp%20Combiview.txt.sha256
53 Human%20Labels.xlsx https://data.nist.gov/od/ds/mds2-2301/Human%20Labels.xlsx
47 https://data.nist.gov/od/ds/mds2-2301/Human%20Labels.xlsx.sha256
53 cluster_assignment_loglik_all.csv https://data.nist.gov/od/ds/mds2-2301/cluster_assignment_loglik_all.csv
47 https://data.nist.gov/od/ds/mds2-2301/Open%20Data%20Challenge%20Notebook%20Human%20Labels%20and%20Plots.py.sha256
47 Readme.txt https://data.nist.gov/od/ds/mds2-2301/Readme.txt
47 https://data.nist.gov/od/ds/mds2-2301/Readme.txt.sha256
47 https://data.nist.gov/od/ds/mds2-2301/Compare%20ML%20Labels.csv.sha256
47 VO2%20-Nb2O3%20XRD%20Combiview.txt https://data.nist.gov/od/ds/mds2-2301/VO2%20-Nb2O3%20XRD%20Combiview.txt
5 Open%20Data%20Challenge%20Notebook%20Human%20Labels%20and%20Plots.py https://data.nist.gov/od/ds/mds2-2301/Open%20Data%20Challenge%20Notebook%20Human%20Labels%20and%20Plots.py
47 https://data.nist.gov/od/ds/mds2-2301/Open%20Data%20Challenge%20Notebook%20Machine%20Labels%20and%20Plots.py.sha256
47 https://data.nist.gov/od/ds/mds2-2301/Condensing%20Write-Ups%20of%20Human%20and%20Machine%20Labeling%20Metrics.docx.sha256
53 Compare%20ML%20Labels.csv https://data.nist.gov/od/ds/mds2-2301/Compare%20ML%20Labels.csv
47 https://data.nist.gov/od/ds/mds2-2301/cluster_assignment_loglik_all.csv.sha256
47 https://data.nist.gov/od/ds/mds2-2301/VO2%20-Nb2O3%20XRD%20Combiview.txt.sha256
5 Open%20Data%20Challenge%20Notebook%20Machine%20Labels%20and%20Plots.py https://data.nist.gov/od/ds/mds2-2301/Open%20Data%20Challenge%20Notebook%20Machine%20Labels%20and%20Plots.py
10 Condensing%20Write-Ups%20of%20Human%20and%20Machine%20Labeling%20Metrics.docx https://data.nist.gov/od/ds/mds2-2301/Condensing%20Write-Ups%20of%20Human%20and%20Machine%20Labeling%20Metrics.docx
0 https://doi.org/10.18434/mds2-2301
47 VO2%20-%20Nb2O3%20Composition%20and%20temp%20Combiview.txt https://data.nist.gov/od/ds/mds2-2301/VO2%20-%20Nb2O3%20Composition%20and%20temp%20Combiview.txt

Tags

  • x-ray-diffraction
  • quantified-uncertainyy
  • combinatorial-methods
  • human-labeling
  • machine-learning-models
  • v-nb-o-thin-films
  • open-data-challenge

Topics

Categories