As we can see in the confusion matrix, it is not the best performing model of all time, but it does a decent job of predicting whether it is spam or not. The model has an accuracy of 79.5% with minimal work in tuning the model. Since this is our baseline model, we are using the results of this to make sure our more advanced model is performing as expected.
The architecture results in 85.3% accuracy, a significant improvement over the naive bayes model (79.5%). The main improvement is actually the reduction of false negatives — tweets that were predicted as non-spam that were actually spam. This hints that the attention mechanism provides meaningful contributions by semantically understanding individual tokens in relation to their surrounding text.