PsyML Toolkit

Origin

I first encountered machine learning through psychology study and behavioral-data analysis. As an undergraduate, I wrote code to process data, run models, understand different algorithms, and examine how they related to psychological data. At the time, the code was simply a tool for learning and research.

Years later, I returned to it. The question was no longer only “How do I run a machine-learning model?” It became “What would a machine-learning workflow look like if it were organized around the practical work of psychology researchers?” PsyML Toolkit developed from that shift.

Today

PsyML Toolkit now brings data review, research-design choices, model comparison, nested validation, interpretation, and reproducibility materials into one local workflow. Version 0.3.0 develops the interpretation side: the results page compares a successful run with a dummy baseline on the same validation, folds and metric and reports fold variability and failures, while optional permutation importance and single-sample SHAP explanations, together with fitted coefficients, describe which variables affect predictions, how one prediction is composed, and the model’s parameters in transformed-feature space. The graphical interface and command line share the same Python core; standalone apps include the runtime, and a quickstart example uses synthetic data to demonstrate training, model saving, and prediction.

Why it matters to me

The project connects my psychology background with research engineering. I want the software to make analysis choices clear, fit the way researchers work, and preserve the information needed to understand and reproduce an analysis. Scientific judgment remains the researcher’s responsibility.

What it does

• Reads nine tabular and statistical-data formats, reviews data locally, and makes variable roles and research-design choices explicit. Data checks report non-missing counts, unique values, category counts and shares, and flag suspected identifier columns without deleting any column.
• Supports classification and regression with preprocessing, model comparison, parameter selection, six validation strategies, and 23 task-specific model options.
• Compares a successful run with a dummy baseline on the same validation, folds and metric, and summarises fold variability and failures; optional permutation importance adds per-validation, per-fold results with signed values.
• When a primary validation is selected and model saving is enabled, saves the final scikit-learn Pipeline; after loading a trusted model, it matches and checks new-data features before batch prediction and export, and can show a single-sample SHAP explanation and fitted coefficients.
• Exports predictions, figures, methods text, warnings, model information, and reproducibility materials.
• Provides Chinese, English, and French interfaces; generated reports are available in Chinese and English.

Before you use it

Input data are processed locally. A final model is saved only when saving is enabled with a primary validation; independent validations have no global final model. Loading joblib or pickle files can execute code, so only trusted models should be opened. Cross-validation and nested validation assess the performance of the training and model-selection procedure; they do not guarantee how the saved model will perform on future data. Generating predictions on new data is not external validation. This version does not provide threshold optimization or automatic retraining.

Permutation importance and single-sample SHAP explanations are approximations based on limited repeated permutations or a finite background; grouped data do not preserve within-subject dependence, and background replacement does not preserve correlations between variables. Neither is a causal explanation or an external performance estimate. Fitted coefficients describe one fit in transformed-feature units, are not converted back to raw units, and include no significance inference.

Researchers remain responsible for study design, validation choices, interpretation, and reviewing automated reports and outputs. The standalone apps are not commercially signed or notarized; your system may ask for confirmation on first launch.

Technical details

Python · scikit-learn · Godot. The analysis core grew from code I wrote during my undergraduate studies; AI assisted development of the current desktop interface and parts of the engineering work. I remain responsible for research design, feature decisions, code review, and interpretation. Code license: Apache-2.0.