MailHeaderParser

Package

Class

Use

Tree

Deprecated

Index

Help

PREV CLASS NEXT CLASS

FRAMES NO FRAMES

SUMMARY: NESTED | FIELD | CONSTR | METHOD

DETAIL: FIELD | CONSTR | METHOD

mailheaderparser
Class MailHeaderParser

java.lang.Object
  mailheaderparser.MailHeaderParser

public class MailHeaderParser
extends java.lang.Object
extends java.lang.Object

MailHeaderParser is the main class that is used to parse e-mail headers of a Columbia University (CUNIX) mailbox that uses IMAP over SSL. It can be modified to work for any mailbox using IMAP over SSL by modifying the server settings.

Constructor Summary
`MailHeaderParser()` Creates a new instance of `MailHeaderParser` and initializes member variables.

Method Summary
`void`	`authenticate()` Authenticates the user.
`void`	`checkDSL()` Checks if the source IP address of the message is statically or dynamically assigned by querying the SORBS DUHL.
`void`	`checkFrom()` Checks whether the domain part of the `From` header agrees with the earliest `Received` host, and accordingly increments either `fromReceivedMatch` or `fromReceivedMiss`.
`int`	`checkSPFExists()` Queries the DNS records for existence of a SPF record belonging to the domain of the earliest `Received` host.
`int`	`checkSPFMatches()` Checks if the `domain` argument's SPF record permits messages to be sent from a host whose address is the `ipAddress` argument.
`boolean`	`checkWellFormed()` Checks whether the current message is well formed.
`void`	`emptyReceived()` Empties `received` headers when a new message is encountered.
`java.lang.String`	`expandBinary(java.lang.String str)` Expands the binary `str` argument to an eight bit binary string.
`java.lang.String`	`expandMacro(java.lang.String inputStr, java.lang.String ipAddress, java.lang.String domain)` Performs macro expansion on the `inputStr` argument according to section 8 of RFC 4408.
`java.lang.String`	`expandMacroTerm(java.lang.String macroTerm, java.lang.String ipAddress, java.lang.String domain)` Expands the `macroTerm` argument as specified by section 8 of RFC 4408.
`java.lang.String`	`getEarliestReceivedHost()` Gets the first host that adds its `Received` header to the message.
`void`	`getMail()` Prompts the user to select one of the folders in the mailbox and retrieves messages inside that folder into `messages`.
`java.lang.String`	`getSourceIP(boolean publicIP)` Gets the IP address of the machine from which the message was sent.
`boolean`	`isCountryCodeTLD(java.lang.String str)` Determines whether the domain component represented by the `str` argument is a country code top level domain.
`boolean`	`isGenericTLD(java.lang.String str)` Determines whether the domain component represented by the `str` argument is a generic top level domain.
`boolean`	`isPrivateAddress(java.lang.String address)` Checks whether the `address` argument is a private address.
`static void`	`main(java.lang.String[] args)` The main entry point into the `MailHeaderParser` class.
`java.lang.String`	`normalize(java.lang.String str)` Removes redundant spaces and other unwanted symbols from the `Received` headers.
`void`	`printReceivedFailure(java.lang.String fromDomain)` Prints relevant message headers of messages to analyze cases where the domain part of the `From` header doesn't match the earliest `Received` host.
`void`	`printReceivedSuccess(java.lang.String fromDomain)` Prints relevant message headers of messages to analyze cases where the domain part of the `From` header matches the earliest `Received` host.
`void`	`printSPFFailure(java.lang.String reason, java.lang.String domain, java.lang.String spfRecord)` Prints relevant message headers of messages to analyze cases where SPF verification against the source IP address failed and also what lead to the failure.
`void`	`printSPFSuccess(java.lang.String reason, java.lang.String domain, java.lang.String spfRecord)` Prints relevant message headers of messages to analyze cases where SPF verification against the source IP address succeeds and also what lead to the success.
`void`	`printStatistics()` Prints relevant statistics to standard output and to the file `cunix_folder_statistics`.
`void`	`processMessages()` Reads messages from the message store one at a time and extracts values from relevant headers.
`java.lang.String`	`reverseMacro(java.lang.String text)` Reverses the representation of the given text splitting at dot boundaries.
`void`	`sortReceivedHeaders()` Reverses the order of the `Received` headers so that they are arranged from earliest to last.
`int`	`spfLookUp(java.lang.String ipAddress, java.lang.String domain)` Performs verification of the `ipAddress` argument against the SPF record of the `domain` argument.
`boolean`	`testIpMatch(java.lang.String ipAddress, java.lang.String ipAddressRange)` Checks whether the `ipAddress` argument falls in the range of the `ipAddressRange` argument.
`void`	`traverse(javax.mail.Folder f)` Traverses the IMAP folder hierarchy recursively to enlist all folders under the default folder in `folderList`

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail

MailHeaderParser

public MailHeaderParser()
                 throws java.lang.Exception

Creates a new instance of MailHeaderParser and initializes member variables.

Throws:: java.lang.SecurityException - if a security manager exists and its checkPropertiesAccess method doesn't allow access to system properties.; java.security.NoSuchProviderException - if a provider for the given protocol is not found.; javax.naming.NamingException - if a naming exception is encountered.; java.lang.Exception

Method Detail

emptyReceived

public void emptyReceived()

Empties received headers when a new message is encountered.

authenticate

public void authenticate()
                  throws java.lang.Exception

Authenticates the user.

Throws:: java.io.IOException - if an I/O error occurs.; javax.mail.AuthenticationFailedException - if an authentication failure occurs.; javax.mail.MessagingException - for other failures.; java.lang.IllegalStateException - if the service is already connected.; java.lang.Exception

printStatistics

public void printStatistics()

Prints relevant statistics to standard output and to the file cunix_folder_statistics.

printReceivedFailure

public void printReceivedFailure(java.lang.String fromDomain)
                          throws java.lang.Exception

Prints relevant message headers of messages to analyze cases where the domain part of the From header doesn't match the earliest Received host.

Parameters:: fromDomain - the reduced domain part of the From header.
Throws:: javax.mail.MessagingException - if a failure occurs when trying to retrieve the header.; java.lang.Exception

printReceivedSuccess

public void printReceivedSuccess(java.lang.String fromDomain)
                          throws java.lang.Exception

Prints relevant message headers of messages to analyze cases where the domain part of the From header matches the earliest Received host.

Parameters:: fromDomain - the reduced domain part of the From header.
Throws:: javax.mail.MessagingException - if a failure occurs when trying to retrieve the header.; java.lang.Exception

printSPFFailure

public void printSPFFailure(java.lang.String reason,
                            java.lang.String domain,
                            java.lang.String spfRecord)
                     throws java.lang.Exception

Prints relevant message headers of messages to analyze cases where SPF verification against the source IP address failed and also what lead to the failure.

Parameters:: reason - the reason that led to SPF verification failure.; domain - the domain currently being looked up.; spfRecord - the SPF Record of the current domain.
Throws:: javax.mail.MessagingException - if a failure occurs when trying to retrieve the header.; java.lang.Exception

printSPFSuccess

public void printSPFSuccess(java.lang.String reason,
                            java.lang.String domain,
                            java.lang.String spfRecord)
                     throws java.lang.Exception

Prints relevant message headers of messages to analyze cases where SPF verification against the source IP address succeeds and also what lead to the success.

Parameters:: reason - the reason that led to SPF verification success.; domain - the domain currently being looked up.; spfRecord - the SPF Record of the current domain.
Throws:: javax.mail.MessagingException - if a failure occurs when trying to retrieve the header.; java.lang.Exception

getEarliestReceivedHost

public java.lang.String getEarliestReceivedHost()
                                         throws java.lang.Exception

Gets the first host that adds its Received header to the message.

Returns:: the first host that adds its Received header to this message.
"by missing" if the earliest Received header doesn't contain "by";
"received missing" if no Received headers exist.
Throws:: java.util.regex.PatternSyntaxException - if the regular expression's syntax is invalid.; java.lang.Exception

getSourceIP

public java.lang.String getSourceIP(boolean publicIP)
                             throws java.lang.Exception

Gets the IP address of the machine from which the message was sent.

Parameters:: publicIP - flag indicating whether a public IP address is strictly necessary.
Returns:: IP address of the machine from which the message originated;
"received missing" if no Received headers are found or none contain the source IP address.
Throws:: java.util.regex.PatternSyntaxException - if the regular expression's syntax is invalid.; java.lang.IllegalStateException - if no match has yet been attempted or the previous match operation failed.; java.lang.IndexOutOfBoundsException - if there is no capturing group in the pattern with the given index.; java.lang.Exception

isPrivateAddress

public boolean isPrivateAddress(java.lang.String address)
                         throws java.lang.Exception

Checks whether the address argument is a private address.

Parameters:: address - the IP address that needs to be checked.
Returns:: true if address argument is a private address;
false otherwise.
Throws:: java.util.regex.PatternSyntaxException - if the regular expression's syntax is invalid.; java.lang.NumberFormatException - if the string doesn't contain a parseable integer.; java.lang.Exception

checkWellFormed

public boolean checkWellFormed()
                        throws java.lang.Exception

Checks whether the current message is well formed. The checks performed include:

whether the Message-Id header is empty.
whether the message originated from a Gmail inbox. Such messages do not contain routing information such as originating IP address and earliest Received host.
whether the message is part of a mailing list. The From header of such messages is typically the e-mail address of the mailing list.
whether the message lacks Received headers or the "by" part of the earliest Received header.

Returns:: true if none of the above conditions are true;
false otherwise.
Throws:: java.lang.Exception

normalize

public java.lang.String normalize(java.lang.String str)

Removes redundant spaces and other unwanted symbols from the Received headers.

Parameters:: str - the Received header to be normalized.
Returns:: the normalized Received header.
Throws:: java.util.regex.PatternSyntaxException - if the regular expression's syntax is invalid.

sortReceivedHeaders

public void sortReceivedHeaders()
                         throws java.lang.Exception

Reverses the order of the Received headers so that they are arranged from earliest to last.

Throws:: java.lang.Exception

isGenericTLD

public boolean isGenericTLD(java.lang.String str)

Determines whether the domain component represented by the str argument is a generic top level domain.

Parameters:: str - the domain component to be checked.
Returns:: true if str is a generic TLD;
false otherwise.

isCountryCodeTLD

public boolean isCountryCodeTLD(java.lang.String str)

Determines whether the domain component represented by the str argument is a country code top level domain.

Parameters:: str - the domain component to be checked.
Returns:: true if str is a country code TLD;
false otherwise.

checkSPFExists

public int checkSPFExists()
                   throws java.lang.Exception

Queries the DNS records for existence of a SPF record belonging to the domain of the earliest Received host.

Returns:: -6 if the ServiceUnavailableException is encountered;
-5 if the InvalidNameException is encountered;
-4 if the NameNotFoundException is encountered;
-3 if the CommunicationException is encountered;
-2 if the earliest Received host has an invalid top level domain;
-1 if the domain hasn't been encountered previously and lacks a SPF record in its DNS entry;
0 if the domain has been encountered previously and lacks a SPF record in its DNS entry;
1 if the domain hasn't been encountered previously and it has a SPF record in its DNS entry;
2 if the domain has been encountered previously and it has a SPF record in its DNS entry.
Throws:: java.util.regex.PatternSyntaxException - if the regular expression's syntax is invalid.; javax.naming.CommunicationException - if the client cannot communicate with the server and a timeout occurs.; javax.naming.NameNotFoundException - if a component of the name cannot be resolved because it is not bound.; javax.naming.InvalidNameException - if the name doesn't conform to the naming syntax of the naming system.; javax.naming.ServiceUnavailableException - if the directory or naming service is unavailable.; java.lang.Exception

expandBinary

public java.lang.String expandBinary(java.lang.String str)

Expands the binary str argument to an eight bit binary string.

Parameters:: str - the binary string that is to be expanded.
Returns:: the expanded eight bit binary string.

testIpMatch

public boolean testIpMatch(java.lang.String ipAddress,
                           java.lang.String ipAddressRange)
                    throws java.lang.Exception

Checks whether the ipAddress argument falls in the range of the ipAddressRange argument.

Parameters:: ipAddress - the IP address to be tested.; ipAddressRange - the range of IP addresses that the ipAddress argument is to be checked against. The range can be either an IP address or an IP address with a network prefix (CIDR notation).
Returns:: true if the ipAddress argument falls in the ipAddressRange argument's range;
false otherwise.
Throws:: java.lang.NumberFormatException - if the string doesn't contain a parseable integer.; java.util.regex.PatternSyntaxException - if the regular expression's syntax is invalid.; java.lang.Exception

reverseMacro

public java.lang.String reverseMacro(java.lang.String text)

Reverses the representation of the given text splitting at dot boundaries.

For example, if the text argument is "aw.bx.cy.dz", the text returned is "dz.cy.bx.aw".

Parameters:: text - the text to be reversed.
Returns:: the reversed text.
Throws:: java.util.regex.PatternSyntaxException - if the regular expression's syntax is invalid.

expandMacroTerm

public java.lang.String expandMacroTerm(java.lang.String macroTerm,
                                        java.lang.String ipAddress,
                                        java.lang.String domain)
                                 throws java.lang.Exception

Expands the macroTerm argument as specified by section 8 of RFC 4408.

The following macro letters are expanded in term arguments:

s - the sender's email address
l - the local-part of the sender's e-mail address
o - the domain-part of the sender's e-mail address
d - the domain-part of the sender's e-mail address
i - the IP address of the SMTP client.
p - the validated domain name of the SMTP client's IP address.
v - "in-addr" if the IP address is ipv4;
"ip6" if the IP address is ipv6.
h - the domain name supplied on HELO/EHLO, normally the domain name of the sending SMTP server.
c - the IP address of the SMTP client.
r - the domain name of the host performing the check, normally the receiving MTA.
t - current timestamp (number of seconds since Midnight, January 1, 1970, UTC).

Parameters:: macroTerm - the macro term to be expanded.; ipAddress - the IP address of the host where the current message originated.; domain - the domain part of the Return-Path header.
Returns:: the expanded macro term if expansion succeeded;
"invalid headers" if the Return-path contains a syntax error;
"null" if PTR query doesn't succeed.
Throws:: java.util.regex.PatternSyntaxException - if the regular expression's syntax is invalid.; java.lang.IllegalStateException - if no match has yet been attempted or the previous match operation failed.; javax.naming.CommunicationException - if the client is unable to communicate with the directory or naming service.; javax.naming.NameNotFoundException - if a component of the name cannot be resolved, because it is not bound.; java.lang.Exception

expandMacro

public java.lang.String expandMacro(java.lang.String inputStr,
                                    java.lang.String ipAddress,
                                    java.lang.String domain)
                             throws java.lang.Exception

Performs macro expansion on the inputStr argument according to section 8 of RFC 4408.

Parameters:: inputStr - the text to be macro-expanded.; ipAddress - the IP address of the host where the current message originated.; domain - the domain part of the Return-Path header.
Returns:: the macro-expanded inputStr if successful;
"invalid headers" if the Return-path contains a syntax error;
"null" if the PTR query doesn't succeed.
Throws:: java.util.regex.PatternSyntaxException - if the regular expression's syntax is invalid.; java.lang.Exception

spfLookUp

public int spfLookUp(java.lang.String ipAddress,
                     java.lang.String domain)
              throws java.lang.Exception

Performs verification of the ipAddress argument against the SPF record of the domain argument.

Parameters:: ipAddress - the IP address of the host from where the current message originated.; domain - the domain-part of the Return-path header.
Returns:: -6 if infinite recursion is avoided due to the SPF record of the domain argument containing itself as an included domain;
-5 if the CommunicationException is encountered;
-4 if the NameNotFoundException is encountered;
-3 if the InvalidNameException is encountered;
-2 if the ServiceUnavailableException is encountered;
-1 if the domain argument doesn't have a SPF record in its DNS entry;
0 if the SPF record for the domain argument doesn't permit messages from the ipAddress argument;
1/10 if the SPF record for the domain argument permits messages from the ipAddress argument.
Throws:: java.util.regex.PatternSyntaxException - if the regular expression's syntax is invalid.; javax.naming.CommunicationException - if the client cannot communicate with the server and a timeout occurs.; javax.naming.NameNotFoundException - if a component of the name cannot be resolved because it is not bound.; javax.naming.InvalidNameException - if the name doesn't conform to the naming syntax of the naming system.; javax.naming.ServiceUnavailableException - if the directory or naming service is unavailable.; java.lang.Exception

checkSPFMatches

public int checkSPFMatches()
                    throws java.lang.Exception

Checks if the domain argument's SPF record permits messages to be sent from a host whose address is the ipAddress argument.

Returns:: -5 if getSourceIP returns "NULL";
-4 if Received headers are missing;
-3 if SPF verification test couldn't be performed;
-2 if the From header's syntax is invalid;
-1 if the Return-path header's syntax is invalid;
0 if the SPF record for the domain argument doesn't permit messages from the ipAddress argument;
1 if the SPF record for the domain argument permits messages from the ipAddress argument;
Throws:: java.util.regex.PatternSyntaxException - if the regular expression's syntax is invalid.; java.lang.IllegalStateException - if no match has yet been attempted or the previous match operation failed.; java.lang.Exception

checkDSL

public void checkDSL()
              throws java.lang.Exception

Checks if the source IP address of the message is statically or dynamically assigned by querying the SORBS DUHL. After performing this check, it increments the relevant counters.

It also outputs the source IP addresses to the file cunix_folder_ping which is later used as input to fping to determine which of the hosts are reachable via ping.

Throws:: java.util.regex.PatternSyntaxException - if the regular expression's syntax is invalid.; javax.naming.CommunicationException - if the client cannot communicate with the server and a timeout occurs.; javax.naming.NameNotFoundException - if a component of the name cannot be resolved because it is not bound.; javax.naming.InvalidNameException - if the name doesn't conform to the naming syntax of the naming system.; javax.naming.ServiceUnavailableException - if the directory or naming service is unavailable.; java.lang.Exception

traverse

public void traverse(javax.mail.Folder f)
              throws java.lang.Exception

Traverses the IMAP folder hierarchy recursively to enlist all folders under the default folder in folderList

Parameters:: f - the folder to traverse.
Throws:: javax.mail.FolderNotFoundException - if folder f doesn't exist.; javax.mail.MessagingException - if a failure occurs while trying to list contents of f.; java.lang.Exception

getMail

public void getMail()
             throws java.lang.Exception

Prompts the user to select one of the folders in the mailbox and retrieves messages inside that folder into messages.

Throws:: java.lang.IllegalStateException - if this store is not connected or this folder is not open when trying to close.; java.io.IOException - if an I/O error occurs.; javax.mail.FolderNotFoundException - if this folder doesn't exist.; javax.mail.MessagingException - if an error occurs while trying to open the folder.; java.lang.Exception

checkFrom

public void checkFrom()
               throws java.lang.Exception

Checks whether the domain part of the From header agrees with the earliest Received host, and accordingly increments either fromReceivedMatch or fromReceivedMiss.

Throws:: javax.mail.MessagingException - if there is an error while trying to fetch the message headers.; PatternSyntaxExpression - if the regular expression's synatax is invalid.; java.lang.IllegalStateException - if no match has yet been attempted or the previous match operation failed.; java.lang.Exception

processMessages

public void processMessages()
                     throws java.lang.Exception

Reads messages from the message store one at a time and extracts values from relevant headers. After reading a message, the following tests are performed:

After performing the above tests and updating member variables for all messages in the selected folder, printStatistics() is called.

Throws:: java.lang.SecurityException - if a security manager exists and its SecurityManager.checkRead(String) method denies read access to the directory.; java.io.FileNotFoundException - if the file does not exist or is a directory rather than a regular file.; java.io.IOException - if an I/O error occurs.; java.util.regex.PatternSyntaxException - if the regular expression's syntax is invalid.; java.lang.Exception

main

public static void main(java.lang.String[] args)
                 throws java.lang.Exception

The main entry point into the MailHeaderParser class. Creates an instance of this class and performs the following tasks:

Parameters:: args - the command line arguments
Throws:: java.lang.Exception

Package

Class

Use

Tree

Deprecated

Index

Help

PREV CLASS NEXT CLASS

FRAMES NO FRAMES

SUMMARY: NESTED | FIELD | CONSTR | METHOD

DETAIL: FIELD | CONSTR | METHOD

mailheaderparser Class MailHeaderParser

MailHeaderParser

emptyReceived

authenticate

printStatistics

printReceivedFailure

printReceivedSuccess

printSPFFailure

printSPFSuccess

getEarliestReceivedHost

getSourceIP

isPrivateAddress

checkWellFormed

normalize

sortReceivedHeaders

isGenericTLD

isCountryCodeTLD

checkSPFExists

expandBinary

testIpMatch

reverseMacro

expandMacroTerm

expandMacro

spfLookUp

checkSPFMatches

checkDSL

traverse

getMail

checkFrom

processMessages

main

mailheaderparser
Class MailHeaderParser