Unconstrained Neural Network

Model overview

Unconstrained neural network classifier using L1 regularization during training followed by post-training weight pruning to achieve sparsity. Unlike parameter-constrained models, this classifier uses ALL available tags as input features, relying on the combination of L1 regularization and pruning to eliminate unimportant connections.

Architecture

The network uses a single hidden layer with 256 neurons:

Input layer: All tags (one-hot encoded)
Hidden layer: 256 neurons with ReLU activation
Output layer: Softmax over all taxonomy classes

Training procedure

Train with L1 regularization (λ=0.0001) to encourage small weights
Apply weight pruning (threshold=0.01) to zero out small weights
The pruned model achieves significant sparsity with minimal performance loss

5 CV folds

62.54% Mean accuracy

0.5898 Mean F1

0.156952 Mean p-adic loss

44,358 Avg non-zero params

90.6% Sparsity

Total train samples: 13,460
Total test samples: 3,365
Hidden layer size: 256
Avg non-zero parameters: 44,358 / 471,509 (90.6% sparse)
L1 regularization (λ): 0.0001
Pruning threshold: 0.01

Cross-validation results

Fold	Accuracy	F1	P-adic loss (mean)	Non-zero params	Sparsity
0	64.19%	0.6070	0.146505	37,289	92.1%
1	63.20%	0.6096	0.126863	38,105	91.9%
2	60.77%	0.5673	0.168868	51,346	89.1%
3	62.91%	0.6002	0.152178	37,979	91.9%
4	61.62%	0.5650	0.190346	57,070	87.9%

Comparison with other models

The unconstrained neural network achieves the best p-adic loss among all models by using more parameters (after pruning), while the L1 regularization and pruning ensure that only the most important connections are retained. This demonstrates the tradeoff between model complexity and prediction accuracy.