A customer recently signed up to a Room Booking System that integrates with Exchange Calendars via EWS (Exchange Web Services). The Room Booking System is a cloud based solution, running on Amazon AWS.
Exchange 2013 is on-premises (separate Client Access & Mailbox servers), deployed in an environment where Exchange Servers are not able to access the internet without going via a Web Proxy. There are no exceptions to this rule.
Ok, the Exchange 2013 Servers are already able to access permitted URLs via the Web Proxy, as Set-ExchangeServer [ServerName] -InternetWebProxy [ProxyServer:Port] is set on each Exchange 2013 server. For example, the Exchange 2013 Servers can communicate with the Microsoft Federation Gateway (this is in place because of federated free/busy calendar sharing).
Inbound EWS Notifications from the Room Booking System succeed, the traffic path is:
Room Booking System > NetScaler Reverse Proxy in the DMZ > CAS Server > MBX Server.
Outbound EWS Notifications fail with MSExchange Web Services Events 6 & 7.
The traffic path for outbound EWS should be:
MBX Server > Web Proxy > Room Booking System
Event ID: 6 / Level: Warning
A notification for subscription [IgBoZGNleG1ieDAwdi5ncm91cC5ob21lZ3JvdXAub3JnLnVrEAAAACGr7pV2bx9NuGpd/bUe1CUbaZ70pj/UCBAAAABDyx/yDHQ4QYJEgZ5uA3Vj] against endpoint [https://the-externalhostedsolution.com/Services/OutlookWS.svc/EWSnotifications] couldn’t be sent.
(Send attempts: 1) Details: WebException: Unable to connect to the remote server Status: ConnectFailure at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context) at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult) at Microsoft.Exchange.Services.Core.NotificationServiceClient.CreateSendNotificationRequestAsync(IAsyncResult requestAsyncResult)
After 9 send attempts, Event ID 7 is logged.
Event ID: 7 / Level: Error
After 9 unsuccessful attempts to send a notification for subscription [IgBoZGNleG1ieDAwdi5ncm91cC5ob21lZ3JvdXAub3JnLnVrEAAAAKtGJU/OTdZOiDx85BshJuCXuprIlT/UCBAAAABDyx/yDHQ4QYJEgZ5uA3Vj] against endpoint [https://the-externalhostedsolution.com/Services/OutlookWS.svc/EWSnotifications], the subscription has been removed. Details: WebException: Unable to connect to the remote server Status: ConnectFailure at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context) at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult) at Microsoft.Exchange.Services.Core.NotificationServiceClient.CreateSendNotificationRequestAsync(IAsyncResult requestAsyncResult)
I looked at the Web Proxy and confirmed the relevant URLs had been whitelisted, and confirmed that anonymous access through the Web Proxy was allowed from the Exchange 2013 Servers.
I watched the real time activity log on the Web Proxy, waited for the next failure to be flagged up, but saw no traffic hit the Web Proxy. Could this be an issue with the Web Proxy logging? No, I fired off a Test-FederationTrust test to confirm the real time activity log was working as expected.
What else can I try? I tried setting the WinHTTP proxy, still no luck. Using Wireshark I confirmed that EWS Notifications were being sent out via the default route, which will of course fail.
So the traffic path for outbound EWS calls is:
MBX Server > Default Gateway [STOPS]
I eventually found that outbound EWS Notifications use .NET web config settings to specify the Web Proxy server.
On the Exchange 2013 Mailbox Servers, I added the following config section to the web.config file:
Note: remove the square brackets around the proxyaddress.
The web.config file is found in the following path:
Save the file, no service restart is necessary, the issue is resolved!