Detect Business Email Compromise Using Generative AI

May 2, 2024

Generative AI is being used lately for generating phishing emails , including generating hyper personalized ones.

How do we recognize such emails ? Generative AI is good at writing emails because it is good at crafting content with given linguistic characteristics. However, Generative AI is good at understanding not just the linguistic characteristics of content but also good at understanding the intent.

That 'intent' of the email is the primary factor in identifying malicious emails.

So can we use Generative AI against generative AI ?

The answer is yes. Nvidia is already advertising such capabilities but that requires a lot of training data to emulate. The good news is that understanding the intent of a text content does not require new training, LLMs are already trained on that.

Here is Detection GPT to prove the hypothesis. We described all the intent characteristics of a malicious email using state of the art prompting techniques and a few examples (no training or fine tuning though).

Example Input Email, masquerading a Support Person to a User

We detected something unusual to use an application to sign in to your Windows Computer. We have found suspicious login attempt on your windows computer through an unknown source. When our security officers investigated, it was found out that someone from foreign I.P Address was trying to make a prohibited connection on your network which can corrupt your windows license key.

If you’re not sure this was you, a malicious user might trying to access your network. Please review your recent activity and we'll help you take corrective action. Please contact Security Communication Center and report to us immediately.1-800-816 0380 or substitute you can also visit the Website: https://www.microsoft.com/ and fill out the consumer complaint form. Once you call, please provide your Reference no: AZ 1190 in order for technicians to assist you better.
Our Microsoft certified technician will provide you the best resolution. You have received this mandatory email service announcement to update you about important changes to your Windows Device.

Example Response from the GPT

Verdict - True. The content is indicative of a business email compromise

1. Urgency: The email creates a sense of urgency by suggesting that an unusual sign-in attempt has been detected and immediate action is required.

2. Lack of Detail: While the email includes some details like an IP address and a date, it lacks specifics about the account or the application used for the sign-in attempt, which can make it difficult for the recipient to verify the claim independently.

3. Attachments: This email does not mention any attachments, which is sometimes a feature in phishing emails, but their absence doesn’t rule out a phishing attempt.

4. Generic Salutation: The email does not address the recipient directly (e.g., "Hello
[FirstName]"), which is common in mass phishing attempts.

5. Unusual Requests: Asking the recipient to contact a "Security Communication Center" via a provided phone number or to visit a generic URL can be seen as unusual. Phishing emails often direct victims to call numbers or visit websites where further fraudulent activity can occur.

6. Spelling and Grammar: There are grammatical errors in the message, such as "a malicious user might trying to access your network," which is not correctly phrased.

The model is is not even GPT-4 for this so the cost of the LLM calls for large number of emails should be very minimal.