Transformer Contribution to URL-Only Threat Detection Under Host-Disjoint Evaluation
Working title, simplified until fully published. The description on this page accurately reflects the actual research, without disclosing implementation-level methodology.
This study takes a more skeptical, targeted look at a question often assumed rather than tested: under a strict evaluation protocol that prevents related URLs from the same host appearing in both training and test data, does adding a compact transformer-based encoder actually improve malicious URL detection over well-tuned classical models? A strong classical approach remained the best overall performer at standard decision thresholds, while the transformer-based approach's clearest advantage appeared specifically at very strict false-positive-rate operating points.
Transformer-based encoders are increasingly applied to malicious URL detection, but their real contribution can be difficult to isolate: public URL datasets often contain duplicate or related infrastructure, and naive random data splitting can let a model implicitly memorize hosts rather than learn genuinely predictive signal. This study asks a narrower, more defensible question: under a strict host-disjoint evaluation protocol, what does a compact transformer encoder actually add on top of strong, well-tuned classical URL-only models?
The evaluated framework combines a compact bidirectional transformer encoder, pretrained on URL character sequences and fine-tuned for classification, with handcrafted lexical and structural features, character- and token-based text models, and tree-based ensemble models. These representations are combined through two alternative fusion strategies: a dense fusion approach, and an approach that treats the internal decision regions of a boosted-tree ensemble as categorical signals. Crucially, the evaluation protocol separates data by host across four distinct roles, representation learning, model selection, fusion training, and final testing, so that no host appears in more than one role.
At standard decision thresholds, a well-tuned classical approach combining lexical and tree-based signals remained the strongest performer overall, with the transformer-fused approach not exceeding it in aggregate. However, at very strict false-positive-rate operating points, the region most relevant to real deployment, the transformer-fused approach showed a clear advantage in detection rate. A closer look at individual predictions showed genuine complementary strengths between the two approaches rather than one uniformly dominating the other.
Taken together, these results support a more qualified view of when transformer-based representations add real value to URL-only threat detection: not as an automatic improvement over well-tuned classical models, but as a way to reshape performance specifically at the stringent operating points that matter most for practical deployment.
Full methodology, results, and the underlying paper are not disclosed here, as this research is intended for conference submission and/or currently under review.