The focus is to combine whole slide imaging (WSI) by optical microscopy and mass spectrometry imaging (MSI) and perform automatic image processing and analysis of these attributes of polished rock sections. The goal is to better understand and empirically predict trace elements in mineral phases screened for petrographic and mineral processing studies. The newly developed methodologies follow a correlative microscopy approach. The approach could be summarised as correlative multi-spatial optical and chemical microscopy (Kamber B., personal communication). It will be used for studying image array outputs of data acquired and reduced in Optical Microscopy (OM), Scanning Electron Microscopy (SEM) and Laser-ablation Inductively Coupled Plasma–Mass Spectrometry (LA-ICP-MS) systems and dedicated software.
Current analytical/statistical workflows are time-consuming and comprise several steps for image spectra acquisition, image processing, management, and image analysis leading to a determined geological application. This project comprises improving data acquisition (1st work package, WP1), developing and implementing software solutions (2nd work package, WP2) for processing and extracting quantitative pixels values within ideally registered image stacks. It uses the correlation microscopy approach that has been trialled on a well-characterized case-study mineral deposit (MBS-4 sample; Murray Brook Pb-Zn VMS deposit, New Brunswick, Canada) and using proprietary software for data acquisition and pre-processing (Element-NIS BR, Nikon; AZtec, Oxford Instruments).
Acevedo M., 2020
The methodological developments aim to generating a reliable database through enhanced data acquisition (WP1) that can be imported to a software platform with an algorithm library for automatic element-mineral association recognition (WP2). The objective of WP2 is to allow users to access whole-slide image pyramids with their underlying quantitative data, leading to more representative studies and interpretations of fossilised geological process at the trace element level. WP1 and WP2 methodologies might permit documenting criptic compositional relationships within spatial context and switching the paradigm from process-centered to data-centered approach for collaborative studies (online WSI platform).
This project uses a variety of programming languages on spectral and imaging Big Data, implementing software solutions to effectively process (data reduction), manage, align (registration) and conveying it to an agnostic front-end user for fluid interaction. Many of these tools already exist in other pioneering fields tapping into whole-slides images processing challenges and we want to learn and bring them to you. This will be a platform with geology-dedicated graphical user interface and scripts will allow having a data-centered approach for better understanding rocks texture and composition.
Specifically, the virtual microscope aims to concentrate all efforts on a common online platform, allowing correlation of image layers for making the best of an outright slow and expensive sample characterization process with dozens of bottlenecks. Afterall, we are all explorers who want to make the best of our data. We understand the collective effort that required hammering or drilling before traversing a few mountains with a precious pack of samples.
Enhancing thin section characterization is the key for linking genetic (scientific) and empirical (3D LeapFrog model) models and assess the intrinsic value of mineral resources with greater confidence levels attending economic and finantial interests. On the analytical side, justifying and carrying on the development of the various forms of mass spectrometry imaging (MSI) have required deep intra-disciplinary knowledge. We expand MSI image processing by:
Combining of microscopy modalities to enhance discovery potential.
Designing a platform where we can elaborate a remote master geological interpretation.
Ultimately reducing primary sampling bias and minimizing downstream analytical uncertainty.
We have some unmentioned collaborators who may be signing up soon. So far, our front members can be recognized for:
Marco A. (Trinity College Dublin, Ireland) has been doing a PhD in this project (2nd year). He is using MSI techniques and OM with polarized light (petrographic microscope). His focus is testing a new paradigm consisting on acquiring whole-slide images with quantitative element-mineral data using research-dedicated proprietary software (ROI) and combining mapping outputs in new software solutions.
Balz K. (Queensland University of Technology, Australia) has been working with a broad range of subjects and state-of-the-art analytical systems, such as LA-ICP-MS and recently TIMA (TESCAN). He is a leading researcher in elemental imaging mass spectrometry, professor in Petrology at Queensland University of Technology, and CI in two pilar of the newly developing "SE Queensland MSI hub of excellence".
Sean McC. (Trinity College Dublin, Ireland) has been working on a paper about Murray Brook VMS deposit (New Brunswick, Canada) and kindly provided a comprehensive database for our developments. He wants to use our techniques to study cobalt (Co) mineral deposits in Ireland since 2021
This section presents 3 research opportunities that are independent, sygergical, and constitute the pillars of this pioneering project. Their development by a broader community will greatly benefit geological studies in the mid-term. A single research group could work on them sequentially from 1 to 3 (see below) for making intrinsic limitations of geological microscopic data clearer in future publications.
Project 1: Implement an advanced virtual microscope for Geosciences
By Balz K.(QUT, Australia)
The digital revolution is transforming the way we view the Earth – from the scale of continents to the atomic arrangement of minerals. Regardless of scale of investigation, novel imaging devices produce multiple ‘maps’ of an area. To find correlative features between these ‘maps’ requires new software. The focus of this project is to develop automatic image processing software for analysis and advanced statistical interrogation of multiple optical and chemical maps of polished rock sections over entire specimen slides (WSI). The goal is to improve and empirically predict trace elements in mineral phases with applications as broad as the search for signs of life in ancient rocks (from Earth and Mars) to energy-critical-element exploration.
To develop a workflow, for use on a personal computer, that will permit the straightforward superposition of optical and chemical maps.
To export the various pixel arrays into a statistical environment where machine-assisted analysis of the data will produce a first-pass interpretation of correlations between chemical an optical features.
To implement in the software, user-friendly tools that enable user-driven advanced statistical interrogation of GB-sized data arrays.
To demonstrate the utility of this workflow with an example relevant to the evolution of microbial life and processing of complex polymetallic ore.
Whole slide imaging (WSI) systems are becoming more stable and detectors are acquiring data at greater sensitivity and speed. This is producing an exponentially growing mountain of data which can currently not be processed with any software. In geological microscopy studies, compositional data acquisition of full rock polished sections at a quantifiable level is yet undeveloped, either using scanning electron microscope based mapping or LA-ICP-MS systems. This gap will be targeted in this project.
The key impact arising from the project is that it will pioneer a personal computer-based approach to combining and aligning multiple ‘maps’ obtained at different spatial resolutions with different distortions, reducing the x-y-z data within the maps and exporting them as a matrix array from proprietary software for advanced statistical analyses that permit implementing algorithms of Artificial Intelligence in the future. Key areas of impact are medicinal chemistry (drug development), search for signs of microbial life, prospecting for energy-critical elements that do not make their own minerals but ‘hide’ as traces in common minerals. This last point is particularly timely in view of Australia’s roadmap to safely producing these elements.
In brief, the advanced virtual microscope aligns with the field of Geochemistry and fundamental research into reconstruction of the origin and evolution of microbial life with applied research into mineral prospecting (industry collaboration opportunities). It also could benefit astronomy efforts to improve the speed of combined imaging techniques in remotely operated rovers on Mars. Contact us for greater insight.
Project 2: Image tile distortion modelling
By Marco A. (TCD, Ireland)
Image processing methods will be use for fixing acquired photomicrographs and element composition maps geometric distortion. These methods have been focused on Digital Histopathology, Biomedicine and Fluorescence Microscopy research, creating a large variety of IT tools/platforms, which could be emulated for geological research. During this time, I ran into Cytomine, QuPath (and ImageJ connection), and the Digital Slide Archive (Girder) software.
After a literature review, I learned about the possibilities of WSI (whole-slide imaging) regarding machine-assisted analysis (AI: ML and DL) of big data. The group of NIST has also advanced their development of WIPP (Web-based Image Processing Pipeline). These breakthroughts have had a great impact for diagnosing cancer or tracking stem cells, and they can become almost as accurate as manual results. The fundamental differences between Digital Histopathology and Geology slide studies are:
The first use scanners that can obtain multi-gigapixel images of hundreds of samples with highly repeatable acquisition conditions, while the second requires to incorporate a polarizer and, optionally, an analyzer in a much larger optical light path. Thus, the second has more “degrees of liberty” on the instrument set up (illumination, WD, lenses, etc.) and you have to consider sample thickness, as it has a strong effect on the VI light path creating varied interference colors in anisotropic minerals (due to light rays traveling at different speeds generating phase differences; e.g.: Michel Levy’s table).
Both progressively apply rigid transformation models between adjacent tiles (in X-Y for 2D and Z for "3D") before stitching, e.g.: TrakEM2 plugin (ImageJ). In fact, biological tissue is soft and easily deformable during sample preparation, i.e.: while sectioning a tumor it can be folded, stretched, holes generation, non-uniform staining, etc. However, sectioning rock samples do not deform the geometry of the slide.
Distortion, i.e.: pixel offset from the real sample texture geometry, is intrinsic to acquisition and very different between techniques (OM, SEM EDX/BSE). Adding more layers into an stitched image stack increases the number of sources of distortion of the extracted pixel data. This was observed surfing the pyramided image stack in QGIS where a single "georeferencing" step also produced a network of badly aligned overlapping areas. This makes multi-modal statistical analysis flawed regardless of the pixel population.
Thus, we have to tackle this problem upstream and for each tile of the set. For rectification image processing techniques (non-rigid transformation models) can be used on the output numeric arrays will allow circunventing doing spatial calibrations of the optical/electron microscopes with metallic gratings observed at a certain magnification range but this can be a tedious and very instrument-specific work.
Proprietary software (e.g.: NIS-Elements BR; MIRA 3 Control Software) should be use only for acquiring the grid of multi-megapixel tiles, thus avoiding the associated computing costs of image stitching ("composition" and "loading") in the online PC that normally causes RAM memory overload. Later on, an open-source solution that is able to stitch rectified tiles producing WSI is required. So, we have already 3 options for stitching and we are left with the task of streamlining any of them with this project. Besides TrakEM2, there is alternative open-source software like the workflow of software solutions showed on the WIPP webpage and the MatLab/Phyton scripts of Ryan Ogliore from Washington University.
In conclusion, as more advanced WSI software is developed, modeling tile distortion will be fundamental for applying a quantitative correlation microscopy approach that is yet untapped in geological studies. More over, working with perfectly-registered pixels in a layer stack will leverage multi-dimensional data and will clearly benefit image segmentation efforts (e.g.: Asmussen et al., 2015; Koch P-H., 2017) and automatic mineral characterization as these advances rely on the number of "feature descriptors" and pixel population respectively. If you know of a research group, software or plugin already working on rock thin sections distortions specific to any of the aforementioned techniques, please share your insight with our group.
Project 3: Quick and accurate SEM-EDX spectral processing (open-source)
By Marco A. (TCD, Ireland)
For whole-slide imaging acquisition, I have been working with the proprietary software AZtec (Oxford Instruments) doing ‘auto PhaseMaps’ and 'QuantMaps' (elements wt.%). However, acquiring large areas of rock sections (cm2) is very computationally expensive and using a binning factor (b.f.) of 1 for the calculations would take 2 weeks for offline processing of an a map, which is unpractical for working in a research laboratory. So, I have been using b.f. = 4, that is averaging 42 = 16 neighboring pixels and sharply decreasing the spatial resolution). This approach has shortfalls but it is the only option nowadays and within laboratory schedules. Thus, if I want to scale up my work and do 4 or 5 thin sections, I need to increase computing power either by novel hardware or software solutions.
Microscopy proprietary software is generally dedicated to run in a single PC central processing unit (CPU). This implies that if you want quicker results, you need to upgrade whole PC and not spare parts (RAM, GPU), which is very expensive nowadays considering Moore's law has declined in the last 7 years (e.g.: NVIDIA CEO talks). For example, AZtec QuantMap offline processing is extremely computationally expensive. It would be more feasible to disable QuantMaps for selection depending on the scanned area.
This project has demonstrated that using a b.f.=4 takes 1 day for completion, whereas downscaling to a b.f.=2 takes a full weekend (thursday afternoon to monday morning) and b.f.=1 is estimated around 2 weeks. In research, pixel quantities must be justifyied with wt.% accuracy, confidence levels, and detection limits. AZtec software do not cover our needs and it is a "black box" with unknown script data structures and patented algorithms. In this context, I made the following questions to Joshua Taillon from NIST, USA (Nov 30, 2019):
Or, should I spend research time figuring out how to use HyperSpy to do the same iterative calculations of pixel wt.%? Will this give me the option to do parallel computing and speed up?
He replied (Dec 3rd, 2019) that HyperSpy API software can do quantification from a spectral image (e.g.: AZtec SmartMap raw data). He believes that HyperSpy is user-friendly with respect to non-pythonic languages with a very comfortable and easy-to-use programming interface to complex high-dimensional datasets. It allows implementing novel methods (i.e. varying samples, processes, etc.) and obtaining results of strong confidence that would be very difficult (or impossible) in routinary vendors' software. Also, it is open-source and portable and can be run on any hardware, whether that's an expensive PCs, or a rented out cluster on AWS, etc. In fact, some operations are parallelizable, and matrix operations in general can be fast, yet HyperSpy is not specifically optimized to be a super-fast analysis tool.
Next, thinking on the expensive option, I need high-performance PC benchmarking measuring AZtec software performance with QuantMap processing. However, this has rarely been shared on online forums and the best information source is careful analysts and collaborators, so we can only trust vendor's PC recommendations and semi-quantitative image array outputs. I have done some benchmarking myself but I also received valuable insight from Tomáš Hrstka (personal communication) working with TESCAN software:
"I did a lot of internal testing on speed and performance and there is a number of factors affecting that. TIMA definitely benefits from big RAM (128GB or more). PCI/M.2. SSD disks with 3000MB/s or more make a big difference compare to classical SSD at 500MB/s. Running the system at one such PCI/M.2.SSD and the hot data on another such PCI/M.2.SSD is ideal. TIMA, unfortunately, does not take the benefit of GPU computing. (I assume GPU will be important for your other applications). Faster CPU you can get the better. Many processes in TIMA are optimized for multithreading but not all, so you want multiple cores, but still decent speed at a single core (e.g. intel Core i9-9900KS @ 4.00GHz). I can only assume that other EDS software will potentially behave similarly. If you have AZtec in you lab you can easily run some TaskManager performance tests to see if multithreading actually works or if GPU is utilized during QuantMap." On January 16, 2020.
Hence, considering a comprehensive literature review and that pixel processing in a GPU would be cheaper and orders of magnitude faster than CPU, we have to consider using the open-source HyperSpy software. The following programmatically steps are proposed:
- Create the phase map in HyperSpy (e.g.: offline phase mapping). This is an improvement from Zanetta et al. (2019), their code could be improved by calculating a mineral phase map directly inside HyperSpy.
- Export the summed spectra for 1 mineral mask to DTSA-II.
- Perform a theoretical calculation and analytical simulation (immitating conditions of a real experiment in the MIRA3).
- Export the used k-factors (this needs testing).
- Parse the files for the k-factor matrices for the X-ray lines I need.
- Create and order a list in alphabetic order.
- Use a rather simple HyperSpy ‘Cliff-Lorimer’ quantification for background-subtracted intensities.
- Repeat for all the pixels of mask.
- Join all the masks’ pixels into one fully quantitative map (boundary pixels would be NaN values from step 0.-).
There are two caveats for testing this approach, the (1) low spectra and (2) sample topography.
(1) Map pixels have low spectra as the individual dwell times are normally less than 1 ms. For example, MBS-4 used 750μm for a 20 hours acquisition time with around 20% dead time at 4μm/px resolution acquiring 340 tiles. Folling Hrstka T. (personal communication), a total of 1K-5K counts/pixel is too low and the minimum could be near 20K depending on the targeted mineral phase. This range will vary depending on the number of peaks (i.e.: characteristic energy lines) and overlaps, for example REE-minerals will need higher spectrum to get decent results. According to Hrstka T., "for a big homogeneous grain, there is almost no difference in EDS quant from a spot and from a raster "summed over the map". On the other hand for small grains reaching the excitation volume of EDS taking one spectrum in the middle eliminating the boundary effect.."
(2) The sample surface topography is a critical factor in quantitative analysis (Newbury and Ritchie, 2015), sometimes as much as composition itself. This also applies for SEM-EBSD analysis where artifacts can be easily created in large portions of the maps. Thus, using the best available technology for thin section preparation will probably decrease the statistical dispersion in EDX maps pixels that was originally thought to originate only from low spectra (i.e.: low signal-to-noise ratio).
To sum up, geological applications using SEM-EDX mapping with "black box" software dedicated to ROI observation (not Automatic Mineralogy WSI) and spot analysis require to scale up and speed up offline processing for obtaining pixel-wise information. Better performance will have an enormous impact on mineral studies as not everybody has access to very expensive QUEMSCAN or TIMA equipment. Researchers have to start looking forward economic extraction (Mineral Processing) elaborating more reproducible and representative studies. This must happen in a context where EDX-SDD detectors are rapidly becoming 10x faster and gaining higher sensibility. I would be looking forward to collaborators for testing this approach or interested in alternatives (e.g.: mixels) that can also be resumed.
Metal Intelligence Network
The project's industrial applications were foreseen early in 2017 within this research network. Its objective has been training and equiping a new generation of leaders in the minerals processing field. Providing lasting novel technological and training methods to build capacity and improve efficiency. Undertaking innovative research across the trans-disciplinary edges of sector compartments.
The long-term applications of this project fall within the framework (see right) occupying the rather new boundaries between three "silo" fields in mineral engineering; Analysis, Processing and Training.
WP3 plans to building a software for advanced mineral analysis from registered compositional layers for allowing implementation of the particle approach of Geometallurgy (Lambert, 2011) with a new level of detail (trace elements).
WP5 goes even further and require having a graphical user interface (GUI), online or offline, for surfing a navigational layer and interrogating the associated compositional layer in the background.
Both developments would permit quicker academy-industry collaboration and tech-enhanced learning about a plant throughput case-study.