Cross-Dataset Generalization and Behavioral Diagnosis for Malicious URL Detection
Working title, simplified until fully published. The description on this page accurately reflects the actual research, without disclosing implementation-level methodology.
This study finds that strong performance within a single dataset does not reliably predict how well a malicious URL detector generalizes to an independently collected dataset, even for a well-designed fused model combining transformer-based and structural representations. A structural signal that was predictive of malicious intent in one dataset showed the opposite association in the other, and increasing how much a transformer model was adapted to its source data improved in-dataset performance while measurably harming its ability to transfer.
Malicious URL detectors are typically evaluated by holding out data from the same dataset used to train them, but strong performance under that setup doesn't establish that the model's decision rule remains valid when the underlying data source changes. This study investigates that question directly, using two independently collected public URL datasets and a rigorous protocol that explicitly accounts for overlap between them.
The clearest finding is a demonstration of how misleading a single "useful" feature can be: one structural signal was strongly predictive of malicious intent in one dataset, but showed the reverse association in the other. A feature that appears highly informative during development can therefore encode dataset-specific patterns rather than a genuinely stable signal of malicious intent.
A second finding concerns model adaptation: increasing how heavily a transformer-based component was fine-tuned to its source dataset improved that dataset's own performance, but came at a real cost to how well the model transferred to the other dataset, and the two datasets showed different patterns of this trade-off.
Finally, controlled behavioral testing, systematically modifying a specific part of the URL and observing how predictions respond, showed that benign and malicious predictions reacted quite differently to the same kind of change, exposing sensitivities that standard accuracy or transfer scores alone would never reveal.
Full methodology, results, and the underlying paper are not disclosed here, as this research is intended for conference submission and/or currently under review.