If you want to troubleshoot the SMTP connection to your outgoing mail server, you can do that by setting the existing debug option in the PHPMailer library, which executes the mail functions in Joomla. This allows you to log the data, mail commands and the connection status to the PHP error log.
Please note that core modifications are not recommended and the following instructions assume that you revert the file to the original version as soon as possible.
Edit the file libraries/joomla/mail/mail.php and add the following lines to the beginning of the function useSmtp(), after line 467:
// debug SMTPmailer
// $this->SMTPDebug = 3; // output data, commands and connection status
$this->SMTPDebug = 4; // low level data output, all messages
$this->Debugoutput = 'error_log'; // output to error log as configured in php.ini
That is all you need to do in Joomla 3.5.0. There is a small additional modification you need to do for Joomla 3.4.8, documented in this article: Debugging SMTP Mail in Joomla 3.4.8
Once you run the mail function, for example by using Mass Mail to a small group of test users including yourself, you can see detailed messages in the PHP error log, for example this failed connection:
[19-Mar-2016 10:47:39 Europe/London] Connection: opening to smtp.gmail.com:587, timeout=10, options=array ( )
[19-Mar-2016 10:47:49 Europe/London]SMTP ERROR: Failed to connect to server: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (10060)
If the SMTP server responds and the connection succeeds, you can see the whole SMTP dialogue in the log file:
[23-Mar-2016 12:47:52 Europe/London] Connection: opening to smtp.gmail.com:587, timeout=300, options=array ( )
[23-Mar-2016 12:47:52 Europe/London] Connection: opened
[23-Mar-2016 12:47:52 Europe/London] SMTP -> get_lines(): $data is ""
[23-Mar-2016 12:47:52 Europe/London] SMTP -> get_lines(): $str is "220 smtp.gmail.com ESMTP t7z282442970wjf.39 - gsmtp
Make sure that you go back to the original version of the file mail.php before the log file starts growing unnecessarily.