How accurate is AntiSpam? Method, data and dated results

Results at a glance

In plain words, for Turkish: of every 100 texts AntiSpam moved to Spam, 98 really were spam; of every 100 spam texts, 96 were caught. The previous engine (version 1.2) caught 70 of 100 on the same data, and misfiled 78 texts in the protected groups instead of 7.

For English: of every 100 texts moved to Spam, 99 really were spam; of every 100 spam texts, 83 were caught. About 1 legitimate text in 180 was wrongly filtered. The English test is built from public research data, so read it with the caveats below.

Definitions

• Precision: of the texts the filter calls spam, the share that really is spam. Low precision means legitimate texts in the Spam folder.
• Recall: of the texts that really are spam, the share the filter catches. Low recall means spam in your inbox.
• False-block rate: of all legitimate texts, the share wrongly sent to Spam. This is the number that matters most for codes and bank alerts, so we report it separately rather than hiding it inside precision.
• Critical false positives: wrongly filtered texts in the protected groups: verification codes, bank and card alerts, delivery updates, appointment reminders and personal texts. We count these individually, not as a percentage, because each one is a real failure.

We don't report "accuracy" as a single percentage. In a dataset that is mostly legitimate texts, a filter that blocks nothing scores high on accuracy and is useless.

What counts as spam

The labels follow the product decision, not an academic convention:

• Spam: scams and phishing, gambling and casino promotions, job, investment, dating and prize bait, and bulk advertising and political texts from senders the recipient never opted into.
• Not spam: transactional texts (codes, bank and card alerts, orders, deliveries), informational texts (appointments, reminders, service notices from companies you deal with) and personal texts.

Counting advertising as spam (a decision made in version 1.4) makes the task harder than scam-only filtering: the line between a bank's promotional text and its service notice is thinner than the line between a bank and a casino. The numbers above include advertising.

The Turkish measurement (October 2026)

Data

13,117 incoming SMS from 2,650 distinct senders, received by the developer between 2018 and October 2026. It is one person's real inbox, so it contains the mix a real user sees: bank alerts, one-time codes, deliveries, school and hospital reminders, carrier notices, personal texts, and eight years of Turkish spam, from illegal betting to fake court summonses. The data is private and is never published or uploaded; the labels and the engine's verdicts are produced locally.

Labelling

Texts were first labelled with simple rules (weak supervision), then every disagreement between the rules, the model and the labels was reviewed by hand. The labelling policy is the one above. An anonymised regression set of about 200 texts, with personal details removed, is kept in the code repository so that every change to the engine is checked against it.

Protocol

Five-fold cross-validation grouped by sender: the texts of any given sender appear in exactly one fold, so the model is always scored on senders it never saw during training. Rules and model are run together, exactly as on the phone, with advertising counted as spam. We prefer this to a random split because spam senders repeat themselves; a random split lets the model memorise a sender's template and then "recognise" it in the test, which inflates every number.

Results

• Precision 98.1%, recall 96.1%.
• 7 critical false positives out of 13,117 texts (codes, bank, delivery, personal), down from 78 with the previous engine.
• Previous engine (1.2) on the same data: recall 70.1%.
• Speed: about 3 milliseconds per text on the phone; the model file is 512 KB and contains weights only, no vocabulary or message text.

What we got wrong before

Our first engines were measured on random splits and looked better than they were. In 2026 we compared lab numbers with the decisions the phone actually logged and found that the shipped filter was missing most Turkish spam sent from alphanumeric sender names, because a rule trusted a regulatory sender code that spammers had started to fake. Engine 1.4 removed that trust, added sender-grouped testing, and made the protected categories explicit rules rather than something the model had to learn. We mention this because a published number is only as good as the test behind it.

The English measurement (October 2026)

English filtering is new in version 2.0. Its numbers come from public data, not from a real person's inbox, and that changes how far you can trust them. Measured on 23 October 2026.

Data

28,675 English texts: 12,827 spam and scam texts and 15,848 legitimate ones. Every one of them comes from public research or open data:

• NCSU smishing set (IEEE Symposium on Security and Privacy, 2024): phishing texts collected from public online SMS inboxes in 2022–2023.
• IMC 2025 smishing set: scam texts reported by members of the public between 2020 and 2024. The links in this set are replaced with a placeholder.
• UCI SMS Spam Collection (2011): an older mix of spam and personal texts.
• Mishra & Soni and SmishX: research sets of smishing, spam and legitimate texts, including a small set of company service notices.
• NUS SMS Corpus: everyday personal texts collected for research at the National University of Singapore.
• An inbox export from India, published on Kaggle under a CC0 public-domain license: bank alerts, one-time codes, delivery notices, promotions and a few scams. We labelled it ourselves with deliberately strict rules, kept only the clear-cut texts and left everything else out.

None of it came from our users, our beta testers or volunteers we asked; all of it was already public. Promotions count as spam here, as they do in the Turkish test.

Protocol

• The model never saw what it scored. Model scores come from five-fold cross-validation grouped by sender, so every text was scored by a model trained without it and without anything else from the same sender. Where a dataset doesn't record senders, near-identical texts are grouped by message template instead and kept in the same fold.
• Rules and model together, as on the phone. The rules run first, exactly as in the app, and the model decides the rest at the same threshold the app ships with.
• Stand-in senders. The rules look at who sent a text, but most public datasets don't say. Each of those texts was given a sender of the kind it would really come from: a phone number for personal texts and most scams, a short code for company notices. The Indian inbox kept its real sender IDs.

Results

• Precision 99.2%, recall 82.6%.
• 87 of 15,848 legitimate texts wrongly filtered (0.55%): 33 by the rules, 54 by the model.

Known weaknesses

• No US, UK or Australian everyday texts in the test. The legitimate bank alerts, codes and delivery notices all come from the Indian inbox. Banks, couriers and government services elsewhere word things differently, and we haven't measured on them yet. To limit the risk, the model only moves a text to Spam when it is very confident, so a borderline text is left alone rather than hidden.
• Delivery notices are the hardest legitimate texts. 6.8% were wrongly filtered. Some of those are really promotions that our labelling filed as delivery notices, so the true rate is somewhat lower, but it is still the weakest spot on the legitimate side.
• Recent scams reported by the public: 77% caught. In the IMC 2025 set the links are replaced with a placeholder, so the engine has to judge those texts on their wording alone, without the link it would normally check.
• Old spam: 47% caught. The UCI collection is from 2011, and its spam reads very differently from today's. We kept it in the test anyway, and it pulls the overall recall down.
• Freshness. Scam templates change monthly. A model trained on older texts catches fewer new templates; the rule updates the app downloads exist for this reason.

What the numbers can't tell you

• Your mix is different. The share of spam, the senders and the languages in your inbox aren't those of the test set. Your precision and recall will differ.
• iOS decides what the filter sees. Texts from your Contacts, from senders you have replied to three or more times, and from iMessage never reach any filter. A test on an inbox can't capture that.
• No head-to-head test yet. We haven't run the same data through competing apps, so we make no comparative accuracy claims. Our comparison page compares approach, scope and price only.
• One developer measured this. Treat it as a documented self-report, not an audit. The reproducible test pack below exists so that others can check.

Reproduce it yourself

Reviewers and journalists can request a test pack by email: 40 anonymised English texts (20 spam and phishing, 20 legitimate including codes, bank alerts, delivery updates and reminders) with the expected verdict for each, plus instructions.

1. Install AntiSpam on an iPhone and select it as the Text Message Filter (setup is described on the main page).
2. Send the texts from a second phone whose number is not in the test iPhone's Contacts, and don't reply to it from the test iPhone (Apple stops filtering a sender after three replies).
3. Check where each text lands: Spam folder, Unknown Senders or the main list. Compare with the expected verdicts.
4. Send us the results, especially the mistakes. Write to iletisim@smsengellemeapp.com.

The pack is the same one we send to publications that review the app, and it is available to anyone who asks.

Measurement history

• 23 October 2026: English engine 2.0 measured on 28,675 texts from public research datasets, model scored out-of-fold: precision 99.2%, recall 82.6%, false-block rate 0.55% (87 of 15,848 legitimate texts).
• October 2026: Turkish engine 1.4 measured sender-grouped on 13,117 texts: precision 98.1%, recall 96.1%, 7 critical false positives.
• October 2026: Turkish engine 1.2 re-measured on the same data for comparison: recall 70.1%, 78 critical false positives.