All about Microsoft Exchange Server Throttling Service and Back Pressure – A geek manual

• What is Exchange Throttling and Back Pressure?
• What are the signs of throttling?
• How can we actually see the current Pressure state?
• What is the throttling impact on mail flow?
• How can we fix backpressure issues temporary-quickly, when long delays are not acceptable by your company or users.

Alright fellows, we’re all once stuck in situations that your mail flow is stopped due to resource throttling in Exchange Servers. In situations like this, you’ll see this kind of messages on any device which is trying to communicate with your Exchange Servers:

4.3.1 Insufficient system resources

And you’ll see emails stuck in queues, waiting to leave your Exchange organization. The reason for all of this is Exchange Throttling Service. Microsoft Exchange Server has a mechanism to monitor system resources, such as Memory Usage, Disk capacity, etc. to calculate server resources pressure, which is called “Back Pressure”.

Backpressure is a system resource monitoring feature of the Microsoft Exchange Transport service that exists on Mailbox servers and Edge Transport servers. Backpressure detects when vital system resources, such as hard drive space and memory, are overused, and takes action to prevent the server from becoming completely overwhelmed and unavailable.

Back Pressure is measured based on Memory and Disk and Queue Lenth (number of messages in queue)

The error “4.x.x Insufficient system resources” that we previously discussed, typically means that Medium or Full Back Pressure is applied in your exchange server. Medium back pressure means that the hardware resources are overused but not that much to stop the whole mail flow. So that Mail from senders in the organization’s authoritative domains can flow. However, depending on the specific resource under pressure, the server uses tarpitting to delay server response or rejects incoming MAIL FROM commands from other sources. That’s why you may see that error on your Smart-Host or Edge Server, and that’s why you will not receive email from the outside world.

There are 3 types of pressure in Microsoft Exchange Server:

1. Normal (low):
   resources are not overused. Mail flow is healthy.
2. Medium:
   The resource is slightly overused. Back pressure is applied to the server in a limited manner. Mail from senders inside the organization is flowing, however, server may reject other sources.
3. High:
   Full back pressure is applied and no mail flow is in Exchange Server.

In a nutshell, Back Pressure is measured based on Memory, Disk and Queue Length (number of messages in queue).

The main metrics of BackPressure is as bellow:

• Hard drive utilization for the drive that holds the message queue database and transaction logs.
• The memory that’s used by the EdgeTransport.exe process.
• The number of messages in the Submission queue.
• The number of uncommitted message queue database transactions that exist in memory.
• Hard drive utilization for the drive that’s used for content conversion.

When Exchange Servers are in BackPressure state, it will notify you by the bellow event logs:

Event log entry for an increase in any resource utilization level

Event Type: Error

Event Source: MSExchangeTransport

Event Category: Resource Manager

Event ID: 15004

Description: Resource pressure increased from Previous Utilization Level to Current Utilization Level.

Event log entry for a decrease in any resource utilization level

Event Type: Information

Event Source: MSExchangeTransport

Event Category: Resource Manager

Event ID: 15005

Description: Resource pressure decreased from Previous Utilization Level to Current Utilization Level.

Event log entry for critically low available disk space

Event Type: Error

Event Source: MSExchangeTransport

Event Category: Resource Manager

Event ID: 15006

Description: The Microsoft Exchange Transport service is rejecting messages because available disk space is below the configured threshold. Administrative action may be required to free disk space for the service to continue operations.

Event log entry for critically low available memory

Event Type: Error

Event Source: MSExchangeTransport

Event Category: Resource Manager

Event ID: 15007

Description: The Microsoft Exchange Transport service is rejecting message submissions because the service continues to consume more memory than the configured threshold. This may require that this service is restarted to continue normal operation.

To easily find the affected Server, I have used a PowerShell Cmdlet which will iterate through all of the Mailbox Servers to find the affected one(s):

Get-MailboxServer | foreach {Get-EventLog -ComputerName $_.Name -Source MSExchangeTransport -LogName Application -After (Get-Date).AddDays(-1) | where {$_.EventID -eq "15004"}}

Now that you know which servers are experiencing Back Pressure, It’s time to fix this. I agree with you all that in the design phase for an Exchange Organization project, hardware resources that are required should have been calculated, but this is not actually happened in many organizations, small to medium businesses, mostly.

Handling memory, is quiet straight forward, because in most cases, you’ll end up adding more resources to your environment, though there are cases that you need to fix software based issues (like when third-party security solution goes crazy and start to eat Memory) to prevent high resource usage or stop it. Except for situations that you may need to balance the load by adding more Exchange Servers into your Exchange Organization, it would not take too much time to accomplish that.

Even though Disk Space can be extended, even easier in a Virtualized Environment, still handling free disk capacity is a bit tricky. Because this is not something you always be able to add more because there are limitations. For example, in an MBR disk, the partition size is limited to 2TB. (4,294,967,295 sectors * 512 bytes/sectors = 2,199,023,255,040 bytes or 2TB). Then sometimes you have to free up space instead of adding it. On the other hand, Microsoft Exchange does not benefit from something like VMWare’s DRS which dynamically prevents you from creating more databases in partitions that there are already databases in them. Therefore you may be stuck in a situation that Exchange mailbox databases are grown enough to fill disk space in a way that Exchange Server throttles the mail flow, and If you are not able to add more space quickly, then, that’s bad!

At this point, NORMALLY there are a few options for you, first, is to move one or couple of your databases to another volume, and the second is to move all mailboxes from one of the databases to another database in another volume or server and then delete the empty database to free up some space. But this will cost a lot of time and what if you don’t have that?

The things that you can do is to first enable “Circular Logging”, especially in DAG databases, to decrease the amount of space that is used by Transaction logs.

If you don’t backup your Exchange servers regularly, then taking a backup is a good idea because Exchange servers will truncate a large amount of transaction logs after backup.

Even though I do not recommend the trick that I’m about to share with you, but there are situations that you’ll be grateful if you know how to simulate a backup without actually doing it to save some time and truncate transaction logs, to free up space.

You can use Volume ShadowCopy Service (VSS) in Microsoft Windows Server, to simulate a finished backup for exchange by running the below commands in a CMD.exe window:

Note: in DAG environments, run these commands in the passive Database server.

C:\Windows\system32>diskshadow.exe #it's located in C:\Windows\System32\ on your server

Microsoft DiskShadow version 1.0

Copyright (C) 2013 Microsoft Corporation

On computer: Behy-MBX03, 9/1/2018 12:39:36 PM



DISKSHADOW> add volume V: (replace "V" with drive letter in which the Exchange database is located)

DISKSHADOW> begin backup

DISKSHADOW> create #It'll take a moment

DISKSHADOW> end backup #This is where it will set the logs to truncate. You will have to

                      # wait for the next log to generate before it starts to happen.

Exchange Server also create a lot of other type of logs, that cleaning them may help you temporary free up a perceptible amount of space.

The bellow PowerShell script can help you achieve that:

$days=7 #amount of days that you want to preserve logs

$ExchangeLoggingPath="d:\Program Files\Microsoft\Exchange Server\V15\Logging\" #this is the full path to your %ExchangeInstallation%\Logging folder



Function CleanExchLogs($TargetFolder)

{

   if (Test-Path $TargetFolder) {

       $Now = Get-Date

       $LastWrite = $Now.AddDays(-$days)

       $Files = Get-ChildItem $TargetFolder -Include *.log,*.blg -Recurse | Where {$_.LastWriteTime -le "$LastWrite"}

       foreach ($File in $Files)

           {Write-Host "Deleting file $File" -ForegroundColor "Red"; Remove-Item $File -ErrorAction SilentlyContinue | out-null}

      }

Else {

   Write-Host "The folder $TargetFolder doesn't exist! Check the folder path!" -ForegroundColor "Yellow"

   }

}

CleanExchLogs($ExchangeLoggingPath)

This usually will give you more than 10GB of space (depending on how recently you cleaned the log files)

In this article we discussed resource throttling, and usual root causes in an Exchange environment. We also have covered the main reason for that, which is usually the disk, and introduced some tricky methods that you can use occasionally, to temporary return the flow to your mail servers to avoid down times in business hours.

I hope this has been informative for you, and I’d like to thank you for reading.