--- title: "B-spline Kolmogorov-Arnold Networks in R: method, validation and an application to PLS-SEM latent scores" output: rmarkdown::html_vignette bibliography: references.bib vignette: > %\VignetteIndexEntry{Method, validation and application} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>", fig.width = 7, fig.height = 3.2) ``` This vignette documents what `splineKANr` computes, how the implementation was validated, and how it can be used after a partial least squares structural equation model (PLS-SEM). The recorded results quoted in Sections 3-5 were produced by the scripts listed in `system.file("REPRODUCE.md", package = "splineKANr")`; the code shown here runs in seconds with the base R backend. ## 1. Model A KAN [@liu2024kan] with widths $(n_0, \dots, n_L)$ composes layers $x^{(l+1)}_j = \sum_{i=1}^{n_l} \phi^{(l)}_{ij}\big(x^{(l)}_i\big) + b^{(l)}_j$. Each edge function is $$ \phi_{ij}(x) = m_{ij}\Big( w^{b}_{ij}\, \mathrm{silu}(x) + w^{s}_{ij} \sum_{m=1}^{G+k} c_{ijm} B_{m,k}(x) \Big), $$ with a mask $m_{ij} \in \{0, 1\}$, base and spline weights $w^b, w^s$ and B-splines $B_{m,k}$ of degree $k$ on an extended knot vector $t_1 < \dots < t_{G+2k+1}$ ($G$ intervals on the domain $[t_{k+1}, t_{G+k+1}]$ plus $k$ knots on each side, as in pykan). The basis follows the Cox-de Boor recursion on half-open intervals $[t_i, t_{i+1})$, $$ B_{i,0}(x) = 1\{t_i \le x < t_{i+1}\},\qquad B_{i,p}(x) = \frac{x - t_i}{t_{i+p} - t_i} B_{i,p-1}(x) + \frac{t_{i+p+1} - x}{t_{i+p+1} - t_{i+1}} B_{i+1,p-1}(x), $$ and its derivative is $B'_{i,p} = \frac{p}{t_{i+p}-t_i}B_{i,p-1} - \frac{p}{t_{i+p+1}-t_{i+1}}B_{i+1,p-1}$. Outside the domain the spline part is, depending on `extrapolate`, zero (`"none"`, pykan's behavior), held constant (`"clamp"`) or continued by its first-order Taylor expansion at the boundary (`"linear"`, default), so that the network stays continuous and has non-zero gradients for inputs beyond the training range. ```{r} library(splineKANr) knots <- kan_make_grid(1, grid_size = 5, spline_order = 3)[1, ] x <- seq(-1, 1, length.out = 201) B <- kan_bspline_basis_r(x, knots, 3, right_closed = TRUE) range(rowSums(B)) # partition of unity on the domain ``` ## 2. Estimation `kan_fit()` minimizes the mean squared error on min-max scaled inputs (mapped to the grid range) and a standardized response, with Adam (either backend) or LBFGS (torch backend). A validation split drives early stopping and the parameters of the best epoch are restored. Grids can be adapted to the data every `grid_update_every` epochs: the new knots are the blend $\epsilon\, t^{\text{uniform}} + (1-\epsilon)\, t^{\text{quantile}}$ of the sample ($\epsilon$ = `grid_eps`), layer by layer, and the spline coefficients are refitted to the old edge functions by least squares with relative ridge penalty `grid_ridge`. Optional regularization is pykan's L1 + entropy penalty on edge activations plus coefficient penalties. Two backends compute the same model: R `torch` (autograd, CPU or GPU) and a base R implementation with analytic back-propagation (`backend = "r"`), which needs no libtorch and serves as the reference for the torch backend. ## 3. Validation The checks distinguish *numerical equivalence* (same inputs and parameters give the same numbers) from *algorithmic equivalence* (the same procedure, with its own random initialization, gives results with the same distribution). **Numerical, components.** The basis agrees with `splines::splineDesign()`, derivatives with `splineDesign(derivs = 1)` and central differences, parameter gradients with finite differences and with torch autograd: ```{r} v <- kan_validate_basis(grid_sizes = c(3, 5), spline_orders = 1:3, backend = "r") v[, c("grid_size", "spline_order", "max_err_splineDesign", "pass")] g <- kan_validate_gradient(widths = list(c(2, 3, 1)), extrapolate = "linear", backend = "r") g[, c("width", "comparison", "max_rel_err", "pass")] ``` **Numerical, pykan.** Fixtures generated with pykan 0.2.8 (float64) are part of the test suite: KANLayer basis, forward pass, gradients and grid update (error < 1e-10); MultKAN networks with up to three layers (forward and gradients < 1e-10); and complete `MultKAN.fit(opt = "Adam")` runs. Recorded results of the training comparison: | Case | Steps | Grid updates | Max. relative loss difference | Max. parameter difference at the end | |---|---|---|---|---| | sin, [1, 1] | 300 | no | 6e-15 | 2e-16 | | interaction, [2, 4, 1] | 300 | no | 1e-14 | 3e-15 | | sin, [1, 1] | 200 | 10, `grid_ridge = 0` | 5e-15 | 4e-15 | | additive, [2, 5, 1] | 200 | 10, `grid_ridge = 0` | 1e-13 | 2e-13 | | sin, [1, 1] | 200 | 10, default `grid_ridge = 1e-8` | 6e-6 | 2e-5 | | additive, [2, 5, 1] | 200 | 10, default `grid_ridge = 1e-8` | 3e-4 | 2e-3 | The only source of difference in the last two rows is the ridge penalty of the refit; it is a deliberate stabilization, and `grid_ridge = 0` reproduces pykan's plain least squares. **Algorithmic.** For three problems with noise standard deviation 0.1, 10 seeds each, both packages ran 500 full-batch Adam steps with pykan's grid schedule from their own initialization. Mean test RMSE (sd): | Problem | splineKANr | pykan | Paired difference (95% CI) | Wilcoxon p | |---|---|---|---|---| | sin, [1, 1] | 0.1005 (0.0013) | 0.1004 (0.0012) | 0.0001 (-0.0000, 0.0002) | 0.43 | | interaction, [2, 4, 1] | 0.1087 (0.0022) | 0.1100 (0.0028) | -0.0013 (-0.0028, 0.0002) | 0.06 | | additive, [2, 5, 1] | 0.1189 (0.0223) | 0.1155 (0.0076) | 0.0035 (-0.0099, 0.0169) | 0.92 | Both packages reach the noise floor (0.1) on the one-dimensional problem and differ by less than 0.002 on average elsewhere; no difference is statistically significant. The larger spread of splineKANr on the additive problem comes from one seed (test RMSE 0.181) that converged to a poorer solution, a reminder that results of a single run depend on the initialization. The pykan default LBFGS optimizer is compared only when the torch backend is available. ## 4. Benchmark Six synthetic problems, 10 seeds, $n_{\text{train}} = 500$, $n_{\text{test}} = 1000$; each tuned method chooses from a small grid on the same inner hold-out. RMSE against the noise-free function (mean over seeds): | Problem | B-spline KAN | Fourier KAN | ANN | GAM | Random forest | OLS | |---|---|---|---|---|---|---| | sin | 0.0052 | 0.0209 | 0.0043 | **0.0011** | 0.0072 | 0.447 | | poly | **0.0019** | 0.0070 | 0.0036 | 0.0034 | 0.0037 | 0.153 | | noisy_sin ($\sigma = 0.2$) | 0.0293 | 0.0696 | **0.0234** | 0.0255 | 0.1203 | 0.447 | | interaction ($x_1 x_2$) | **0.0058** | 0.0253 | 0.0070 | 0.3343 | 0.1094 | 0.334 | | additive | 0.0109 | 0.0463 | 0.0143 | **0.0070** | 0.1085 | 0.625 | | Friedman 1 ($p = 10$) | 1.207 | 1.569 | **0.570** | 1.394 | 2.044 | 2.441 | The B-spline KAN is the most accurate method on the polynomial and the interaction problems and close to the best on the smooth univariate and additive ones, where a GAM, which matches the true structure exactly, is better. It is consistently and clearly more accurate than the Fourier KAN comparator under the same protocol. On Friedman's ten-input problem the ANN is about twice as accurate and the KAN results vary strongly between seeds (sd 0.35): with one hidden layer of width 3 chosen from the small tuning grid, the KAN is not flexible enough for this target. The KAN is also the slowest method here (base R backend, 3-18 s per fit). ## 5. Application: predicting satisfaction and loyalty from PLS-SEM scores A common workflow in marketing and management research estimates a measurement and structural model by PLS-SEM [@hair2022primer] and then asks whether the linear structural equations miss non-linear effects. Below, a simulated customer survey (two exogenous constructs with five 10-point items each and a satisfaction construct with three items) is analyzed with the base R PLS-PM routine shipped in `inst/applications`, and an additive KAN is fitted to the latent scores. ```{r} src <- system.file("applications", "pls_pm.R", package = "splineKANr") source(src) set.seed(42) n <- 600 quality <- rnorm(n) image <- 0.5 * quality + sqrt(0.75) * rnorm(n) sat <- 0.4 * quality + 0.9 * pmax(image, 0) + rnorm(n, sd = 0.4) items <- function(lv, k, prefix) { z <- sapply(seq_len(k), function(i) 0.8 * scale(lv)[, 1] + 0.6 * rnorm(length(lv))) out <- pmax(pmin(round(5.5 + 1.6 * z), 10), 1) colnames(out) <- paste0(prefix, seq_len(k)) out } survey <- data.frame(items(quality, 5, "q"), items(image, 5, "im"), items(sat, 3, "s")) blocks <- list(Quality = paste0("q", 1:5), Image = paste0("im", 1:5), Satisfaction = paste0("s", 1:3)) paths <- rbind(c("Quality", "Satisfaction"), c("Image", "Satisfaction")) pls <- pls_pm(survey, blocks, paths) pls_path_table(pls, "pls") round(rbind(CR = pls$cr, AVE = pls$ave), 3) ``` ```{r} sc <- pls$scores train <- seq_len(400) kfit <- kan_fit(sc[train, c("Quality", "Image")], sc[train, "Satisfaction"], hidden = integer(), grid_size = 5, epochs = 300, backend = "r", seed = 1) lfit <- lm(Satisfaction ~ Quality + Image, data = as.data.frame(sc[train, ])) test <- as.data.frame(sc[-train, ]) rbind( kan = kan_metrics(test$Satisfaction, predict(kfit, test[, c("Quality", "Image")], backend = "r")), ols = kan_metrics(test$Satisfaction, predict(lfit, test)) ) ``` ```{r, fig.height = 3} op <- par(mfrow = c(1, 2)) for (v in c("Quality", "Image")) { plot(kan_partial_dependence(kfit, sc[train, c("Quality", "Image")], v), main = v, ylab = "PD of Satisfaction") } par(op) ``` In this simulation `Image` raises satisfaction only when it is above average (a kinked effect), which the linear structural model can represent only by a single slope. The partial dependence of the KAN recovers the kink (flat below the mean, rising above it) and the out-of-sample error is somewhat lower than that of OLS. The curve for `Quality`, whose true effect is linear, bends slightly in the tails where there are few observations; such wiggles should not be over-interpreted. The gain is not automatic: with half the sample size and a milder non-linearity the same comparison favored OLS, because the extra flexibility costs variance. Whether a KAN improves on the structural equations is an empirical question to be settled out of sample, as below. For honest predictive assessment the PLS weights must be estimated on the training rows only; `inst/applications/run_application.R` does this inside every fold of a repeated cross-validation. Recorded results on three public customer data sets: the ECSI data `seminr::mobi` (n = 250) and `plspm::satisfaction` (n = 250), and the corporate reputation data `seminr::corp_rep_data` (n = 344; formative drivers, Hair et al. 2022). 10 x 5-fold cross-validation; RMSE of the standardized target score, mean over repeats; "PLS-SEM" is OLS on the target's predecessors in the structural model, all other methods use every antecedent construct: | Data, target | PLS-SEM | B-spline KAN | GAM | ANN | Random forest | Fourier KAN | |---|---|---|---|---|---|---| | mobi, satisfaction | **0.584** | 0.616 | 0.592 | 0.598 | 0.623 | 0.657 | | mobi, loyalty | **0.753** | 0.776 | 0.770 | 0.792 | 0.753 | 0.832 | | satisfaction, satisfaction | **0.567** | 0.601 | 0.599 | 0.590 | 0.587 | 0.630 | | satisfaction, loyalty | 0.723 | 0.722 | **0.710** | 0.719 | 0.718 | 0.767 | | corp_rep, satisfaction | **0.852** | 0.864 | 0.860 | 0.869 | 0.899 | 0.898 | | corp_rep, loyalty | **0.667** | 0.682 | 0.679 | 0.678 | 0.714 | 0.720 | In these data the linear structural model predicts as well as or better than every flexible method, and the B-spline KAN performs like a GAM or an ANN. With 250-350 respondents and latent scores that are linear combinations of Likert items, there is no detectable non-linearity that pays for the extra parameters. Here the KAN is useful as a diagnostic: its additive partial-dependence curves give a direct visual check of the linearity assumption of the structural equations, and its cross-validated error shows that relaxing that assumption does not improve prediction. The same pipeline is meant for data where non-linear effects (thresholds, saturation) are expected, or for larger samples. ## References