Speaker
Description
Finding the optimal workflow to clean and prepare data, train a machine learning (ML) model, select between different models, and evaluate the final model properly is a complex task. Novice users can easily be overwhelmed by the variety of available methods for the individual workflow steps like data splitting, metric selection, and model evaluation. Many tools have been created to support specific workflow steps (e.g. hyperparameter tuning), often by automation. However, the current landscape lacks a comprehensive tool that guides researchers with their individual research problems through all ML workflow steps efficiently [1].
Our new software mlguide aims to fill this gap by providing a user-friendly, interactive platform that facilitates the selection of suitable methods at each stage of the machine learning process. Users can define input characteristics of their dataset (e.g., sample size and task type) and their research question (e.g., whether they want to compare models, evaluate a selected model, or both). Based on this information, mlguide recommends suitable methods for each workflow step, e.g. whether a simple train-test-validation split or nested cross-validation is more suitable for the concrete ML task. These recommendations are backed up by a growing database of evidence from scientific literature.
mlguide is developed in the context of KI-FDZ, a project that investigates the AI readiness of the German Health Data Lab (HDL; in German Forschungsdatenzentrum Gesundheit, FDZ). Specifically, mlguide will be a central component of the so-called “AI Sandbox”, a user-friendly AI toolbox that aims to allow testing specific user scenarios in a secure processing environment with data protection-compliant data sets.
The mlguide toolkit consists of three modules. 1) The guidance engine, which is part of the R package mlguide, handles user requests and derives suitable methods. 2) The knowledge base, mlguide.core, stores evidence from scientific literature in a structured format. 3) The graphical user interface, the R Shiny application mlguide.app, allows users to specify their research question and select methods for their research problem. For now, mlguide can handle supervised regression and classification tasks on tabular data. Due to its modular design, future extensions to other task and data types are easily possible.
Currently, a small team of ML experts is reviewing scientific publications and extracting results in a structured format to fill our knowledge base mlguide.core. As this is a labor-intensive process, we performed initial experiments on using large language models (LLMs) to support the evidence extraction process. Our early results in this regard are promising, but we expect human validation of the LLM output to remain necessary to ensure an accurate evidence extraction. In addition to the current capabilities and architecture of mlguide, we present the status of our LLM experiments and share our vision for an efficient, LLM-enhanced evidence extraction process.
References
[1] Detjen, H. et al.; Designing Machine Learning Workflows and Experiments with Ease: A Scoping Review of Interactive Tools. Under preparation. 2024.
