-->

Sunday, June 4, 2017

Exchange Set Retention Policy In Bulk For Specific Mailbox Type

We recently rolled out a new Retention Policy in my Exchange 2016 environment but I only wanted it to apply to Linked and User Mailboxes; not Shared or Resource Mailboxes.
We'll set a different policy for those. So I needed a way to apply this new Policy to a certain mailbox type. We'll get this done quickly with PowerShell!
 
Generally, when you apply a Retention Policy in bulk you would run Get-Mailbox | Set-Mailbox without any filters, and that would apply it to all Mailboxes in the organization.
 
We don't want that, so we'll use a filter to grab the Mailbox Type.
 
Fire up the Exchange Management Shell and run the following cmdlet:
 
Get-Mailbox -Filter {(RecipientTypeDetails -eq 'UserMailbox')} | Set-Mailbox –RetentionPolicy "UserMailbox Retention Policy"
 
The above cmdlet will apply your policy named "UserMailbox Retention Policy" to User Mailboxes only.
 
Since this is a Resource Forest, I also have Linked Mailboxes in my environment. So we'll need to apply the Policy to those as well:
 
Get-Mailbox -Filter {(RecipientTypeDetails -eq 'LinkedMailbox')} | Set-Mailbox –RetentionPolicy "UserMailbox Retention Policy"

This cmdlet will filter only Linked Mailboxes and apply the "UserMailbox Retention Policy"

**Note** Change "UserMailbox Retention Policy" to the name of your policy.

If you have multiple policies for different mailbox types, change the RecipientTypeDetails -eq 'mailboxtype' to one of the following:

UserMailbox

LinkedMailbox

SharedMailbox

RoomMailbox

EquipmentMailbox

For instance, setting a Room Mailbox Policy would look like so:

Get-Mailbox -Filter {(RecipientTypeDetails -eq 'RoomMailbox')} | Set-Mailbox –RetentionPolicy "RoomMailbox Retention Policy"

Now your different mailbox types will have the proper policies applied!

Saturday, June 3, 2017

Exchange Export List of Mailboxes Created After a Certain Date

In my organization, we have a billing structure for each site, where they charge back resources like mailboxes to the company. This is done monthly, so I needed a way to get a list of mailboxes to send to site managers.

We'll use our old buddy PowerShell to export a CSV of mailboxes that were created after a certain date.

Fire up the Exchange Management Shell (EMS) and run the following:

Get-Mailbox -ResultSize unlimited | where {$_.whenmailboxcreated -gt (get-date "4/18/2017")} | select displayname,whenmailboxcreated | Export-CSV C:\Temp\Mailboxes-4-18-17.csv

What the cmdlet does is grabs all mailboxes created after 18APR17 with "-gt" (greater than) operator and then exports the CSV with the mailbox display names and when they were created.

**Note** Change the date to how far you need to go back and change "C:\Temp\Mailboxes-4-18-17.csv" to the path and filename of your choosing.

Now you have a when were these mailboxes created report to send out!

Saturday, May 27, 2017

Exchange Messages Missing From The MessageTrackingLog

In my Exchange 2016 environment, we run an Edge server, which is where I generally do message tracking; this gives a better picture of message flow, since all external mail goes in or out of the Edge.

If you don't run an Edge server, you'll need to do message tracking on your Mailbox Servers.

The issue,is, just running the Get-MessageTrackingLog on one Mailbox server, searches only on that server, which means some messages that you know went out, are missing from the log.

So, we need a way to search the log on every server we have. PowerShell makes it easy!

Fire up an Exchange Management Shell on one of your Mailbox Servers, or on your Exchange Management Tools machine and run the following:

$Servers = Get-ExchangeServer;  $Servers | where {$_.isHubTransportServer -eq $true -or $_.isMailboxServer eq $true} | Get-MessageTrackingLog -recipients email@address.com -start 05/23/2017

**Note** Change email@address.com to the email addy you're working with and change the -start date

Now, your message tracking searches will show all messages not just the ones local to a server.

Exchange Count the Number of Messages In The MessageTrackingLog

I had to diagnose a mailbox forwarding to external issue recently, and while I could tell that messages were going out by using Message Tracking on the Edge server, the user that had the forward wanted a count of how many messages were being sent.

So this is quick post on using PowerShell to get a count of messages by sender in the MessageTrackingLog.

Fire up the Exchange Management Shell on the Edge server if you have one, or on one of your Mailbox Servers.

And run the following commands:

$count = get-messagetrackinglog -Recipients "email@address.com" -Start 5/23/2017

$count | Group-Object -Property Sender | Select-Object name,count | sort

**Note** Change "email@address.com" and -start date

You can also do -sender instead of -recipient in the first variable, like so:

$count = get-messagetrackinglog -Sender "email@address.com" -Start 5/23/2017

And if you want to narrow down the results further, you can also add an -EventID like so:

$count = get-messagetrackinglog -Recipients "email@address.com" -EventID SEND -Start 5/23/2017

Happy message counting!

Sunday, May 21, 2017

Exchange/AD Export List of Users With Empty Email Address Fields

In my Exchange 2016 environment, I run a Resource Forest, which means we have to use MIM to sync users from the Accounts Forest. The way we have it configured, the attribute that triggers a sync is the Email Address field. If that field is empty, the account won't get created in the Resource Forest.

Not every account needs an email address, or sometimes the provisioning script might not populate the email address.

My bosses wanted an "empty email address" report, so I needed a way to export a list of users with empty email address field, and their respective OU's.
Since this is an Accounts Forest (with no Exchange) we'll have to use Active Directory PowerShell cmdlets...which aren't as easy to use to grab recipient information.

So, here's a quick one-liner that will grab all ADUsers with no email address populated; the full path of the Organizational Unit they live in; and export that list to a CSV file.

Fire up Active Directory Module for Windows PowerShell and run the following cmdlet:

Get-ADUser -Filter {EmailAddress -notlike "*"} -Properties EmailAddress | Select-Object Name,@{n='OU';e={$_.canonicalname -replace "/$($_.cn)",""}} | Export-Csv "C:\Temp\EmptyEmailAddresses.csv"

**Note** Change the "C:\Temp\EmptyEmailAddresses.csv" path to wherever want to save the csv.

Your csv output will look like so:

Name OU
Stacey Branham exchangeitup.com/Mailboxes
User1 exchangeitup.com/NonMailUsers
User2 exchangeitup.com/NonMailUsers
User3 exchangeitup.com/DisabledUsers
User4 exchangeitup.com/DisabledUsers

Happy reporting :)

Sunday, May 7, 2017

Exchange Count Total Number Of Items In All Mailboxes

After completing the user mailbox migration from Lotus Notes to my Exchange 2016 environment, our higher-ups wanted a "how much email did we move?" report.
While it does seem kind of arbitrary, they wanted a full count to go along with their return on investment report for the board.

We used the Quest migration tool to move the mailbox items, and while it is a pretty decent tool, it didn't really give a full picture of items moved, without digging through each migration batch log - which would take forever.

So I created two quick scripts to count the number of all items in each mailbox in the Exchange organization, and then present the total.

Get Item Count By Database

The first script will count items in each Mailbox Database, which can be useful for a more granular breakdown.

You can download the Get-TotalItemCountInDBs.ps1 file on my gDrive.

Or copy and paste the following text into Notepad, and save as a .ps1 file:

 ##Change "DBNAME" to the Database you're working with
$Mailboxes = Get-MailboxDatabase "DBNAME" | Get-Mailbox

 $MailboxTotalItemCount = 0
 foreach ($Mailbox in $Mailboxes)
 {
 $MailboxStats = Get-MailboxStatistics -Identity $Mailbox
 $MailboxItemCount = $MailboxStats.ItemCount
 $MailboxTotalItemCount = $MailboxTotalItemCount + $MailboxItemCount
 }
 Write-Host "Total Item Count:      $MailboxTotalItemCount"


**Note** Since this is just a quick script, I didn't have to time to make it pretty and prompt for the DB name, so you'll need to replace the "DBNAME" with the database in the ps1 itself, before running it.

When the script is done running, you'll get an output like so - this is for one of my databases:

[PS] C:\Users\stacey\scripts>.\Get-TotalItemCountInDBs.ps1 
Total Item Count:      2983548

Get Item Count In All Databases

The second script will crawl through every database, and present the total count for all mailboxes on all databases.

You can download the Get-TotalItemCount.ps1 on my gDrive

Or, once again, copy the text below and save as a .ps1:

 $Mailboxes = Get-MailboxDatabase | Get-Mailbox
 $MailboxTotalItemCount = 0
 foreach ($Mailbox in $Mailboxes)
 {
 $MailboxStats = Get-MailboxStatistics -Identity $Mailbox
 $MailboxItemCount = $MailboxStats.ItemCount
 $MailboxTotalItemCount = $MailboxTotalItemCount + $MailboxItemCount
 }
 Write-Host "Total Item Count:      $mailboxTotalItemCount"


**Note** You don't have to specify any database in this script, just run it as is.

When this script is done (and yes it will take a while) you'll get the results like so:

[PS] C:\Users\stacey\scripts>.\Get-TotalItemCount.ps1
Total Item Count:      15797733

Now, you'll have a number to present to your "handlers" showing that: yes, we moved over 15 million items (in my case) over to Exchange; and yes that was overkill since most users won't ever even know how to find most of those items :)

Sunday, April 23, 2017

Exchange Internal Mail - 400 4.4.7 Message Delayed

In my Exchange 2016 environment I have three multirole servers in a DAG, running on VMWare. Everything had been humming along nicely until everyone started getting the following bounce message:

4/17/2017 9:09:53 PM - Server at MBX1 (10.28.69.162) returned '451 4.4.0 Target host responded with: 4.4.7 Message delayed'

This was for internal messages, between the three Exchange servers. Which is pretty odd, since they're set up to transfer mail out of the box.

I also found the following Events on the servers:

Log Name:      Application
Source:        MSExchangeTransport
Date:          4/17/2017 5:20:52 PM
Event ID:      1035
Task Category: SmtpReceive
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      MBX1
Description:
Inbound authentication failed with error UnexpectedExchangeAuthBlobCheckForClockSkew for Receive connector Default MBX1. The authentication mechanism is ExchangeAuth. The source IP address of the client who tried to authenticate to Microsoft Exchange is [10.28.69.163].


As you can see, the "ClockSkew" part clues us in what's going on.
The problem was that the Time sync on the VMware hosts wasn't set up correctly, so the system clocks on the Exchange guests were off...way off.

Anything over 5 minutes will generally cause issues, with everything from mail routing to DAG replication.

To fix it, follow my previous post here on fixing the VMWare host time issues.

Saturday, April 8, 2017

Exchange Resource Forest Selective Trust - Part 2

Continuing from Part 1, we'll now set up our authentication permissions in the Resource Forest to allow users in the Accounts Forest to connect to the Exchange servers.

Setting Authentication Permissions

1.Open Active Directory Users and Computers (ADUC).

2.Under View, ensure that Advanced Features is selected.

3.In the console tree, click the OU where your Exchange Servers live.

4.Right-click the Exchange server object that you want users in the Accounts Forest to access, and then click Properties.

5.On the Security tab, do the following:

Click Add.


In "Enter the object names to select", type the "Resource Forest Exchange Auth" group name, and then click OK.

Select the Allow check box next to the following permissions:


Read
Allowed to Authenticate
Read Account Restrictions
Read DNS Hostname Attributes
Read MS-TS-GatewayAccess
Read Personal Information
Read Public Information permission

Then click OK.

6. Do the above steps on each of your Exchange servers.


Depending on the size of your environment, allow time for the permissions to replicate across forests.

**Note** In my environment, we have some "screwy" ACLs, so we had to add those perms on the Resource Forest DC's as well, just in case you hit a wall and no one can authenticate even after applying the perms.

Now choose one of the users in the "Accounts Forest Exchange Auth" Group and have them try to connect or open Outlook - they should be allowed.

Then choose a user not in the group and do the same - they should be blocked from access.

Now, you should be ready to go and secure with your Selective Trust all set!

Exchange Resource Forest Selective Trust - Part 1

My environment utilizes two Resource Forests (each holding an Exchange 2016 organization) one in the US and one in Europe; both Resource Forests share one Accounts Forest.

We needed to lock down authentication to the US forest because of business regulations...meaning no user in Europe would be able to connect to the US Exchange servers.

In order to fulfil this requirement, I implemented a Selective Trust which is:


"Selective authentication over a forest trust restricts access to only those users in a trusted forest who have been explicitly given authentication permissions to computer objects (resource computers) that reside in the trusting forest."

In this two part series, I'll show you how to enable and configure the Selective Trust and how to apply authentication permissions on the Exchange Servers in the Resource Forest.


Create Security Groups

In order to keep user authentication clean and manageable, we'll use security groups in both forests.

In the Accounts Forest create a Security Group called something like "Accounts Forest Exchange Auth" and add the users who will be allowed to authenticate with the Exchange Forest.

**Note** The users in this group are the only ones will be able to connect to their Linked Mailboxes; users not in this group will get errors in Outlook that it can't find their mailbox.

In the Resource Forest create a Security Group called "Resource Forest Exchange Auth" and add the "Accounts Forest Exchange Auth" Security Group from the Accounts forest into that group.


**Note** This will basically be a nested group where the Accounts Forest Group is a member of the Resource Forest Group. This way, we only have to add members to one group, one time.

We're going to use this group in the Resource Forest to apply authentication permissions on the Exchange servers later.


Enabling Selective Authentication

Using the GUI:

1. Open Active Directory Domains and Trusts.

2. In the console tree, right-click the domain node for the forest root domain, and then click Properties.

3. On the Trusts tab, under either Domains trusted by this domain (outgoing trusts) or Domains that trust this domain (incoming trusts), click the forest trust that you want to administer, and then click Properties.

4. On the Authentication tab, click Selective authentication, and then click OK.

Using the command line:

Open an elevated CMD, and run the following (all on one line):


Netdom trust TrustingDomainName /domain:TrustedDomainName /SelectiveAUTH:Yes /userD:DomainAdministratorAcct
/passwordD:DomainAdminPwd

**Notes**
TrustingDomainName
The DNS name of the trusting forest root domain in the trust that is being managed.

TrustedDomainName
The DNS name of the forest root domain that is trusted in the trust that is being managed.

DomainAdministratorAcct
Your domain admin account

DomainAdminPwd
Your domain admin password

Now your Selective Trust has been created.

In the next article, we'll set the permissions on the Exchange servers, to allow users to authenticate.

Saturday, April 1, 2017

Exchange Create Address List for Equipment Mailboxes

In my Exchange 2016 environment, users weren't happy with having to search or scroll down the GAL to find resources such as cars and projectors. They were used to the way Lotus Notes would present these. The easiest way to deal with this is to create separate lists for Resource Mailboxes.

I'll show you how to use PowerShell to create a Resources/Equipment Address List.

Fire up the Exchange Management Shell (EMS) and run the following cmdlet:

New-AddressList -Name "All Resources" -RecipientFilter {((RecipientType -eq 'UserMailbox') -and (RecipientTypeDetails -eq 'EquipmentMailbox'))}

**Note** You can name the list whatever you like. I chose "All Resources" because it falls in line with default lists like All Rooms, and that's what my bosses wanted ;)

What this cmdlet does is, create a new Address List (which is dynamic so any new Resource Mailbox is added automagically) that pulls in mailboxes that match the "Equipment" recipient type details.

Next, go ahead and force an Address List update, by running:

Get-AddressList | Update-AddressList

**Note** Depending on how many lists you have and their size, it might take some time to complete.

After the update finishes, Outlook in online mode will pick it up immediately; cached mode will pick it up after it syncs.

Outlook Address List

Sunday, March 26, 2017

Exchange Set Equipment Mailbox Reservation Duration In Bulk

In my current environment, we migrated from Lotus Notes to Exchange 2016, and some users don't like the changes that come along with it, especially when dealing with booking Resource Mailboxes.

The default maximum duration to reserve an Equipment Mailbox is 24 hours in Exchange. It makes sense for meeting rooms, but most users will need resources such as cars for longer than a day, which results in them having to book several appointments on the Equipment Mailboxes.

We're going to change the max duration to 7 days (to give users a little more leeway) and we're going to change it for all Equipment Mailboxes at once.

This is really easy with PowerShell!

Fire up the Exchange Management Shell (EMS) and run the following cmdlet:

Get-Mailbox | where {$_.recipienttypedetails -eq "equipment"} | Set-CalendarProcessing -MaximumDurationInMinutes 10080

**Notes**

- I set the value to 10080 which is 7 days.

- The default value is 1440 minutes (1 day).

- The maximum value you can enter is 2147483647 (which is 40 centuries, in case you're wondering).

- You can set a value of "0" which is unlimited - prolly don't wanna do that since other users couldn't then book the resource, unless you've allowed overlapping bookings.

- For recurring meetings, the value applies to the length of an individual meeting instance.

Now your users can schedule those long road trips reserve a projector for a conference in one shot!



Saturday, March 25, 2017

Exchange Convert Linked Mailbox to MailUser

In my current Exchange 2016 environment we run two resource forests, which means we have to sync accounts and contacts by using MIM 2016 (Microsoft Identity Manager).
Early on in the configuration, some Linked mailboxes were created in the wrong forest, and those needed to be MailUser accounts instead.

How I found this out was: mail from people in my forest was being delivered to the user's mailbox (which was mistakenly created) in my forest instead of the user's primary mailbox in the other forest.

To convert it, we'll need to break the link from the mailbox to the master account, disable the mailbox, and enable the MailUser.
This has no affect on the primary mailbox, so it's safe.

You can't do the first step in the EAC, so we'll be using PowerShell...which is faster and way more fun anyhow!

Fire up the Exchange Management Shell (EMS) and run the following cmdlets:

First, we'll remove the mailbox-to-master account link:

Get-Mailbox "mailbox name" | Set-User -LinkedMasterAccount $null

Next, we'll disable the mailbox:

Disable-Mailbox "mailbox name" -Confirm $false

Now, we'll enable the MailUser:

Enable-MailUser "user name" -ExternalEmailAddress emailaddress@otherresourceforest.com

**Note** You'll need to change "mailbox name", "user name", and "emailaddress@otherresourceforest.com" to match your environment.

And there you have it; mail will be delivered to the user's mailbox in the other forest.

If I have time, I might make this into a script...so stay tuned! Or if any of you readers wanna do it, post it up in the comments :)

Exchange 2016 Event ID 1035 "Inbound authentication failed with error"

After installing CU4 on my Exchange 2016 servers and bouncing them, they started throwing MSExchangeTransportDelivery Event 1035 every few minutes.

Here's the full error:

Log Name:      Application
Source:        MSExchangeTransportDelivery
Date:          3/19/2017 10:30:14 AM
Event ID:      1035
Task Category: SmtpReceive
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      MBX1.exchangeitup.com
Description:
Inbound authentication failed with error UnexpectedExchangeAuthBlobCheckForClockSkew for Receive connector Default Mailbox Delivery MBX1. The authentication mechanism is ExchangeAuth. The source IP address of the client who tried to authenticate to Microsoft Exchange is [127.0.0.1].


The error, in this case tells you exactly what to look for. Check the system clocks across your Exchange servers and Domain Controller.

For now (to get Exchange running properly), manually set the clocks the same and then restart the Frontend Transport and Transport Services on each Exchange server.

Now, figure out why your clocks are wrong. If these are VM's, follow my post on how to set your VMWare host and guest clocks.
If they're hardware machines, prolly need to replace the CMOS battery, which prolly means if it's that old, you need to replace the server itself ;) 

Thursday, March 16, 2017

Exchange PDF Attachments Sent Externally From Outlook Are Garbled

In our ongoing Lotus Notes to Exchange 2016 migration, we've had reports of outgoing PDF attachments from Outlook being garbled on the recipient end, which was confusing because they would work fine coming from Notes. It was hit or miss (not affecting every recipient) so it was a little tough to track down.

The received PDF would look like so:

Garbled PDF Attachment

Turns out, the solution was pretty easy (once I realized what was happening).
The migration tool carried over the contacts from Notes to the users' Outlook, and it seems that some contact entries were corrupted.

The Fix:

Have the user delete and recreate the contact, if they have it in their contacts list.

-or-

Have the user clear the recipient from their autocomplete entry, and manually type in the email address and resend the message with the PDF.

Sunday, March 12, 2017

Exchange 2016 Shell Error "WinRM cannot process the request; XML contains a syntax error"

A while back I wrote a post about the EWS API causing trouble on one of my Exchange 2016 servers.
Well, I needed to bump up the VM resources on my Exchange servers, which requires a reboot. After the reboot, the remnants of the EWS app came screaming to life. This left the Exchange Management Shell useless, which also left the server in maintenance mode, because I couldn't use the shell to run the maintenance cmdlets.

Tip: if you're in a bind, just use the Windows PowerShell and load the Exchange snap-in by running: Add-PSSnapin Microsoft.Exchange.Management.PowerShell.SnapIn;

The EMS tried to connect to each Exchange server before finally giving up and throwing the following error:

Connecting to remote server "servername" failed with the following error message: WinRM cannot process the request because the input XML contains a syntax error.
FullyQualifiedErrorId: -21447108477

Exchange Shell XML Error


The Fix:

I tried tons of things and what I finally was, the system clock on my Exchange VMs were off; not by a lot, but enough to cause errors.

If you're running Exchange on VMWare, follow my post of setting the VM host time correctly.

Sunday, March 5, 2017

Exchange Restrict Room Mailbox Booking For A Certain Group

We recently migrated from Notes to Exchange 2016, and while most users are happy with the way Exchange/Outlook works, some don't want to change and like the way Notes used to handle certain things; like who has access to book meetings rooms. The problem is: it creates more work because as the Exchange administrator, I have to now individually manage each room instead of letting the booking delegates do it, but oh well :)

Luckily with PowerShell, settings booking permissions is simple!

First, create a Security Group and add the members who will be allowed to book a room. Yes, you can create a distribution group, but I prefer to use security groups when assigning permissions to anything...I'm also OCD about keeping my AD/Exchange clean ;)

Once you have your group built, lets call it "Room 1 Allowed" run the following cmdlets in the Exchange Management Shell (EMS):

Set-CalendarProcessing "Room 1" -AllBookInPolicy $false

Set-CalendarProcessing "Room 1" -RequestInPolicy $none

Set-CalendarProcessing "Room 1" -BookInPolicy "Room 1 Allowed"

**Note** Change "Room 1" to the name of your Room Mailbox.

To see the permissions on the Room calendar, run:

Get-CalendarProcessing "Room 1" | fl

You should see that the Security Group you added in the "BookInPolicy" is in there.

Now, have a member of the group try to book the room and it will be successful. And have someone not in the group try, and they'll get a warning message saying the meeting was rejected because they aren't allowed.

Saturday, February 18, 2017

Exchange 2016 Room Mailbox Calendars Only Showing Busy

We recently migrated from Lotus Notes to Exchange 2016 and after moving Rooms and Equipment Mailboxes over, some of the calendars didn't keep their permissions.
When users went to schedule a room they wouldn't see who had booked the room, they would only see "Busy" like in this example:
 
Room Calendar No Details
 
This was strange because I left the default settings of AddOrganizerToSubject and DeleteSubject set to "True" so users should be able to see the organizer of the meeting.
 
After some digging around, I found that the Mailbox Folder Permissions for the "default" user were set  to "none"...but not on every Room mailbox. Looks like the migrator tool wasn't able to keep those permissions intact.
 
To view permissions on a Mailbox fire up the Exchange Management Shell and run:

Get-MailboxFolderPermission "room 1:\Calendar"

**Note** Change "room 1" to the room mailbox name in your environment.

In my case permissions were like so:

FolderName     : Calendar
User                 : Default
AccessRights   : {None}
Identity            : Default
IsValid             : True

Luckily with PowerShell, this is an easy fix. We're gonna set permissions on all Room Mailboxes, so we don't have to go digging through each of them.

**Note** You might get a bunch of yellow warnings saying no changes were made on some Mailboxes; that's ok, it just means the perms were already set.
 
In the (EMS) and run the following cmdlets:
 
$r = Get-Mailbox -RecipientTypeDetails RoomMailbox
 
$r | %{Set-MailboxFolderPermission $_":\Calendar" -User Default -AccessRights LimitedDetails}
 
**Note** I chose "LimitedDetails" because I don't want users seeing everything about meetings in case they might be sensitive like "we're closing down your office". You can set different levels of access, which are listed in this TechNet article.

Now open a Room Calendar in Outlook or OWA, and you should see the Organizer and Room:

Room Calendar With Details

 Now you'll have happy users!

Sunday, February 5, 2017

Exchange Veeam Backup Timeout - VSSControl: Failed to prepare guest for freeze timeout 900 sec

We're using the Quest tool to migrate a ton of mailboxes from Notes over to Exchange, and since it uses quite a bit of resources on the Exchange servers, our Veeam backups were failing. This led to a massive amount of transaction logs building up in Exchange, which led to more backup failures. Round and round we went, never getting a good backup.

The error in Veeam was:

“VSSControl: Failed to prepare guest for freeze, wait timeout 900 sec."

The issue was that Veeam was timing out because it couldn't freeze the guest VM (the Exchange mailbox server) in order to start the backup. After the 900 seconds (15 minutes) expired, Veeam would give up and fail.

What I had to fix it was adjust a registry setting on the Veeam server, to lengthen the timeout threshold.

On the Veeam server, open regedit and navigate to:

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication

Add a DWORD (32-bit) named VssPreparationTimeout

**Note** The value is in milliseconds (decimal), the default timeout is 900000, which equals 15 minutes

We're going to start by setting it to 30 minutes, so set the value to 1800000

**Note** Even after 30 minutes, mine kept failing so I had to keep bumping it up in increments until I found the sweetspot...which for me was 66 minutes.

After you create the DWORD, close regedit.

Stop any running jobs or wait for them to complete (which they won't), close the Veeam control panel, then restart the Veeam Backup Service.

Open the Veeam control panel up, and start your backup job.

Now it should freeze the guest successfully, and get a good backup!

Exchange/Active Directory Add Proxy Addresses in Bulk

In my current project, we're migrating from Lotus Notes to Exchange 2016 in a Resource Forest, and we needed to do some manual mail routing. That involved setting proxy addresses in our Accounts Forest.
 
In my previous post I went over how to set the targetAddress with ADModify.
 
This time we'll be adding entries to the proxyaddresses by using PowerShell.
 
I used two methods of bulk adding proxies; by Organizational Unit (OU) and by importing a CSV (our OU structure is a mess, and we had some users scattered about so I used a CSV list to finish up).
 
We're going to add a proxy address to the already existing ones.
This new proxy will be an accepted domain in our Exchange environment, but not the authoritative one. We'll use something like @test.domain.com. We basically want to be able to route mail from Notes to Exchange, using a temporary address.

My organization's email address is the common format of firstname.lastname@domain.com, so the cmdlet will pull the GivenName (first name) and Surname (last name) of each user, and stick a "." in the middle, while adding the test accepted domain to the end.

To bulk add proxies by OU, fire up the Active Directory PowerShell, and run:

Get-ADUser -Filter 'Name -like "*"' -SearchBase 'OU=US,DC=exchangeitup,DC=com' -Properties proxyaddresses | % {Set-ADUser $_ -add
@{proxyAddresses="smtp:"+ $_.GivenName + '.' + $_.Surname +"@test.exchangeitup.com"}}

**Note** You'll want to change 'OU=US,DC=exchangeitup,DC=com' and test.exchangeitup.com" to match your environment.

**Note** You'll also notice that the "smtp:" is lowercase; that is by design because we want it as an alias, not primary address.

After the cmdlet runs, check over a few users AD attributes and you should see the newly added proxyaddresses entries.

Now, we'll see how to import a CSV list of users and set the proxies.

First, create a CSV in the following format.

SAM,SMTP
username1,user1@test.exchangeitup.com
username2,user2@test.exchangeitup.com
username3,user3@test.exchangeitup.com
username4,user4@test.exchangeitup.com

You'll have one column, titled SAM, and one titled SMTP; with the usernames (samaccountnames) and accepted domain email addresses listed.

Save it to wherever you're going to run AD PowerShell from, and name it something like proxies.csv - specific, I know :)

Now, fire up the AD PowerShell and run the following:

import-csv proxies.csv | foreach {Get-ADUser $_.SAM | Set-ADUser -add @{proxyaddresses = "smtp:"+($_.smtp)}}

This cmdlet will add the proxy addresses (as an alias with the lowercase "smtp:") using the samname and the other accepted domain (test.exchangeitup.com) we'll be using for our routing.

Once again, go check a couple users' AD attributes and you'll see the newly added proxies.

Happy migrating!

Friday, February 3, 2017

Exchange 2016 100% CPU and Event ID 5011 "MSExchangeServicesAppPool"

This post is mostly for my own reference, but just in case you come across the same issue, I'll outline the problem and the fix.

We recently migrated several hundred users from Notes to my Exchange 2016 CU2 environment and shortly after the migrations, one of my three mailbox servers was getting constantly hammered.

The CPU was at 100% usage, performance was terrible, and it even caused my Kemp load balancer to flap the virtual services on this server every 30 seconds or so. In particular the MAPI and EWS services.

That led me to start checking what was up with EWS.

The System Event log was full of the following error:

Log Name:      System
Source:        Microsoft-Windows-WAS
Date:          1/28/2017 3:11:50 PM
Event ID:      5011
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      EXCH16-MBX1
Description:
A process serving application pool 'MSExchangeServicesAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '18076'. The data field contains the error number.


I remembered that one of our admins was testing out an EWS Retention policy script and had installed the Exchange Web Services Managed API 2.2  on this box.

So, I uninstalled the API 2.2 and bingo! Problem solved; CPU was back to normal, all client connections were now stable.

As a workaround, we installed the EWS Managed API 2.2 on our DAG witness server (since it has the Exchange tools installed on it) and now run the script from there.

It works!

Saturday, January 28, 2017

Exchange Set TargetAddress Attribute in Bulk With ADModify

My current project involves migrating from Lotus Notes to Exchange 2016, and since we didn't pay for a full suite of a certain migration utility, we had to do some mail forwarding the old fashioned way: setting the targetAddress attribute on the AD accounts.
The targetAddress is basically a forwarding address that routes mail to a different mail environment...in this case from Notes to Exchange.

In our cutover, I had to set the target for 800 users in bulk, so I chose to use AdModify. You could write a PowerShell script to do this. but why reinvent the wheel? :)

This article assumes you already have ADModify downloaded and know how to use it.
If not, here's an overview.

We're going to set the 'targetAddress' to same as the 'mail' attribute; this will forward our mail to the proper location - from Notes to Exchange.

Once you have ADModify loaded up and you have your OU's and users specified, navigate to the Custom Tab:

ADModify Custom Tab
 
 
Put a check next to "Make a customized attribute modification"
 
In the Attribute Name, enter: targetAddress
In the Attribute Value enter: %'mail'%
 
This will set the target to same value as mail. You can use other attributes, like mailnickname if you have that populated.
 
**Note** You  must use the percent sign % and apostrophe ' at each end of the mail attribute name.
 
Next, hit the Go! button.
 
ADModify targetAddress to mail
 
 
Review successes and errors.
 
Now you can check ADUC (Active Directory Users and Computers) on one or a few of the accounts you modified.
 
Open Properties of the account, navigate to the Attributes Tab and scroll down to "targetaddress". The new address matching the "mail" address should be listed.
 
Happy migrating!