<img height="1" width="1" src="https://www.facebook.com/tr?id=1529264867168163&amp;ev=PageView &amp;noscript=1">
blog_listing_hero_img.jpg

Intelligently Using Migration Endpoints to Speed up Migrations to Exchange Online - Part 3

This is a multi-part article in which we will cover Migration Endpoints. First, we will cover what Migration Endpoints are, what you use them for and how you can manually configure a migration endpoint. In the second part of this article, we will dive deeper into how you can leverage multiple migration endpoints to potentially speed up your migration to Exchange Online. Lastly, we'll discuss some of the most common mistakes regarding Migration Endpoints and how to avoid or solve them.

Part 3: Troubleshooting Migration Performance and Migration Endpoints


Nothing can be more frustrating than trying to move mailboxes to Exchange Online, only to be stopped by some (seemingly random) error... Although there are many things that can affect a mailbox move, we’ll focus on the following two of the three areas that often cause problems. It’s not that these issues happen frequently. But if an issue related to mailbox moves occurs, it’s often related to one of the two following areas:

  • There is a "generic" issue with MRS (a.k.a. the onboarding service; MRS in Office 365)
  • Migration endpoint related failures (e.g. the endpoint is not responding)
  • Slow mailbox moves (e.g. moves take many hours where they should've taken 'minutes')

For now, we'll leave MRS issues "out of scope". As you will learn later in this article, the troubleshooting steps to investigate a failed or slow/stalled move are just as applicable to this scenario as well.

Migration Endpoint Failures

Over time, error messages in Office 365 have become a lot better at describing what the actual issue is. I remember times where pretty much anything that could go wrong with hybrid was reduced to a generic error message: "Exception has been thrown by the target of an invocation." Today, luckily, not so much anymore.

In terms of mailbox moves, any issue pointing in the general direction of connectivity can often be brought back to issues with the migration endpoint. When you are facing a connectivity error, chances are that the GUI will not even allow you to create the migration batch: while creating a batch, a remote connection test is performed to ensure the endpoint is working as expected.

One of the errors you might encounter is "The connection to the server '<fqdn>' could not be completed."

Before pointing to the networking guys (we love to do that!), remember that there’s a few things that need to happen in Exchange before a migration endpoint will start to work. First, figure out what server(s) your migration endpoint is pointing to, and then verify if the MRSProxy is configured for those servers using the following PowerShell cmdlet:

mvh1

Note that the MRS Proxy is normally configured when running the HCW. However, sometimes the wizard cannot update the MRS Proxy settings leading to you having to manually enable the component using the following PowerShell cmdlet:

MVH2-1

How do you know if this worked? There’s a cmdlet that you can execute from Exchange Online PowerShell: Test-MigrationServerAvailability. Please take a moment to remember this cmdlet: it will be your best friend in troubleshooting (remote) endpoint issues.

The The syntax to manually test a migration endpoint is the following:MVH3-2

In the above example, you need to specify the FQDN and credentials yourself. This can be useful when also testing for credential problems. The credentials you enter should belong to an account that has sufficient permissions to move mailboxes to Office 365. Although this can be a member of the Organization Management role group, the Recipient Management role group is a better option (following the principle of least privilege, that is). Note that assigning the Recipient Management 'management role' is not sufficient. It should be the Recipient Management 'role group'. A mistake that can be made easily but will prevent the test cmdlet to execute successfully (as will the mailbox moves, FWIW).

Despite the ability to manually enter the required information, I strongly suggest starting by using the credentials that are currently used for the migration endpoint you are trying to test; that way you can rule out credential issues. If you don’t have the credentials, you can test the existing endpoints as-is with the following command:

MVH2-1

The output of the command will look like the one below (failed one). For sake of brevity, I omitted a few lines from the output. When looking at the output, the “Message” and “ErrorDetail”-properties are the most interesting ones anyway:

MVH5

As you can see from this example, the first few lines of the ErrorDetail property contains a lot of usefor information. From the message, we learn that the connection could not be completed. We can also see that the call to the endpoint failed because “no service was listening on the specified endpoint.” When this happens, it is a good time to talk to your networking guys: the test wasn't even able to open a connection with the MRS Proxy, and that often points to a networking issue 😉

Like I said before: the error messages are quite descriptive and typically point you in the right direction. Don’t skip on running the Test-MigrationServerAvailability cmdlet, as it can save you a lot of time!

Before moving onto migration performance, let’s get back to the credentials of the endpoint. If your mailbox moves don't start, or the endpoint test fails mentioning something about an account not having sufficient permissions, or a “401 Unauthorized” error message (like below), chances are that the password of account that is configured on the endpoint has expired (or the account has expired/been locked). Again, the Test-MigrationServerAvailability cmdlet is your best friend in testing the existing credentials, and again once you’ve tested new credentials:

MVH6

In this example, the answer lies in the bold text above: “the remote server returned an error: (401) Unauthorized.” This message generally rules out a connection issue, as the test was able to successfully connect to the remote server and present the configured credentials to the MRS Proxy. When you run into this scenario: run the test again with an updated password, or a different account and see if that makes a difference. If so, you can update the endpoint with new credentials after testing.

In general, if connectivity to your migration endpoints is configured correctly and you’re not suffering from intermittent connectivity issues, there’s very little that will go wrong with an endpoint. However, if you are having connectivity issues, it can sometimes be daunting to figure out whether it’s the migration endpoint, firewall, reverse proxy, switch, or anything else in between Office 365 and your Exchange server(s). As such, heed the advice from earlier posts: keep your connectivity simple!
In this example, the answer lies in the bold text above: “the remote server returned an error: (401) Unauthorized.” This message generally rules out a connection issue, as the test was able to successfully connect to the remote server and present the configured credentials to the MRS Proxy. When you run into this scenario: run the test again with an updated password, or a different account and see if that makes a difference. If so, you can update the endpoint with new credentials after testing.

Troubleshooting Migration Performance

Almost every time I talk to someone that suffers from (supposedly) slow migration performance, they are quick to blame Microsoft because they believe Microsoft is (actively) throttling them. Typically that is not true in all but the most extreme cases. Most of the times, the issue isn’t with Microsoft but their own environment.

Why is that? Well, Microsoft has no interest in actively throttling your mailbox moves INTO Office 365. Given that a migration is often a first impression of their service, they better make sure it's a good one. After all, you only get a single shot at a first impression! Unfortunately, this doesn’t mean there aren't some other conditions that may lead to throttling of some kind. 

One thing you will be subjected to occasionally is the resource-based throttling. This is when some resource constraint in Office 365 causes a move to stall or slow down (significantly) and is usually depicted by information which you can find in the Move Request’s statistics. To get to this information, use the Get-MoveRequestStatistic cmdlet:

Move for mailbox '/o=ExchangeLabs/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=<value>' is stalled because DataMoveReplicationConstraint is not satisfied for the database 'EUPRD04DG001-db007' (agent MailboxDatabaseReplication). Failure Reason: Database edbf0123-2f3a-4567-8910-bb4a78a9080b doesn’t satisfy constraint SecondDatacenter. There are no available healthy database copies. Will wait until 4/15/2018 1:27:53 AM.

In the above example, there seems to have been an issue with mailbox database replication across Microsoft’s datacenters, therefore pushing back on the influx of new data. There’s other reasons you might be throttled too, including issues with High Availability, Content Indexing, Individual Server Resources (CPU,...), etc.

While there are many things that can influence the velocity of a mailbox move, and in the occasion that you suffer from low performance, you should ask yourself the following question: “how slow is too slow”?

The answer to that question isn’t always easy to find. From a personal perspective, you might want mailboxes to move within matter of minutes. In reality, you are confined to the least performing link in your migration chain. For example, if you have a 100Mbps Internet connection, you cannot exceed that connection’s upload speeds for any migration to Office 365. The same is true for the performance of your source environment (e.g. disk subsystem, etc).

Before pointing the finger at Microsoft, consider what your weakest link in the chain might be, and then check whether the performance of the mailbox move matches what that component can sustain. If your migration speed is below what one would commonly expect in such scenario, you should start looking into what's wrong.

The first thing to establish poor performance is to measure it. Although it’s a few years old, the following article contains an invaluable script (and information on how to run it), so you can measure the performance of your current and past move requests.

Once you’ve collected the migration metrics, you can troubleshoot accordingly. Once you have collected the metric, start looking into the move request statistics. Below is an example of how you can retrieve some useful information:

MVH7



Note that, even when you have created Migration Batches, you will see individual move requests being spun up. Alternatively, the same information is exposed using the Get-MigrationUserStatistics cmdlet.

The output of the command above, will look like this. I omitted quite a bit of redundant and irrelevant text, to keep things clean:

MVH8

The output in the example above is from a move request that completed successfully. Nonetheless, the migration report contains a TON of information on the move, the phases within the move, actions it completed and – in case of a failure – which failure it encountered.

Using this information, along with the metrics from earlier you should be able determine whether the issue is on-premises or in Office 365. The good news is that if the issue is with Microsoft, there isn’t much you can do other than to open a call with Microsoft and hope for them to “fix” the issue quickly. However, some there is no issue that can be fixed "on the fly" like a move that is stalled due to a High Availability constraint. In such case, you will often see moves automatically continue after having sat idle for a while...

If the issue is with your own environment, there’s plenty to verify: the performance of your source environment, networking, etc.

I hope you've enjoyed this "series" of articles, and feel free to reach out if you have any follow-up questions or remarks!

Get started with Mailscape 365