Public Folder Migration

Is drive shipping best for large PST migrations to Microsoft 365?

Posted on 25/04/202222/07/2022 by Alethia Erchen

Do you need to migrate thousands of PST files, really big mailboxes or large email archives to Microsoft 365 (and is drive shipping the answer?)

What is a PST file?

A PST file is an Outlook Data File (.pst) that can be used to store messages and other Outlook items. From the ‘outside’ they look like a single file, but inside they have the structure of a pseudo mailbox.

Larger organisations in particular can have a legacy of many thousands of PST files sitting on individuals’ local drives, home drives and file servers. They would historically get created as a way of staying within mailbox quota (when Exchange mailbox quotas on premises where perhaps not as generous as they are online). PST files were also used as a convenient way of preserving mailbox contents – perhaps after an individual had left the company. In fact PSTs used in these scenarios are widely considered a pest – especially when organisations switch to the cloud.

PST files can also be used as an interim store when migrating between platforms, as many mail systems can export emails and attachments into PST format, and conversely, import content from a PST file. It is this ‘use-case’ that is the subject of this particular article…..

What is drive shipping?

Microsoft’s drive shipping service is multi-step process via which large amounts of data can be uploaded into Microsoft cloud via an interim device which is physically shipped to a Microsoft location.

This is how it works:

Instead of transferring data across the network, the technique involves writing PST files to a hard drive along with a mapping file (for example, migrate PST <filename> to <username> primary or archive mailbox).

The (encrypted) hard drive is then physically shipped to a designated Microsoft location from where data centre personnel pre-stage the contents into Azure. The files are then ingested into Exchange Online according to the supplied mappings.

If you have TBs of emails you want to move into Microsoft 365, drive shipping via PSTs is an option open to you.

This Microsoft article goes into all the different iterations of what you need to do and the cost of using this service.

The same technique can be used to perform migration of mailboxes that are over 100GB in size or if the user’s mailbox contains one or more messages that exceed the 150-megabyte (MB) message limit (in which case resorting to PST files is recommended).

Using interim PSTs is also an option for migrating the contents from large email archives such as Enterprise Vault and EMC SourceOne, or email journals form platforms such as Mimecast.

The question is: Is drive shipping PSTs a good option for your large email archive migration?

Here are the pros and cons:

Pros

It’s low cost…on paper*. The cost to import PST files to Microsoft 365 mailboxes using drive shipping is $2 USD per GB of data.
It minimises impact on your network: if you have a sub-optimal network that cannot handle large amounts of data being transferred, using a data drive to ship PSTs to Microsoft negates any network concern. Even on networks that can support around 500Mbs you can experience slow performance when you start to drive a large migrations alongside regular user activity.
It avoids the impact of Microsoft throttling:Microsoft applies throttling to avoid overloading its servers. Although you won’t experience the effect of throttling when using native Microsoft mailbox moves to migrate your mailboxes, many email archive migration solutions use the EWS protocol to move your data, and this protocol is subject to throttling, although Microsoft has made throttling easy to ease off during the course of a bulk migration.

Cons

*It can work out expensive:At face value, $2 USD per GB is cost-effective, for example, a 20TB project would be $40,960 to ‘drive ship’, but this does not include the added overheads of getting your data onto the drives (see next point).
PST preparation is labour-intensive. Suffice to say that manually extracting data from archives into PST files and then preparing them for upload can be super time-consuming. Native tools for extraction out of third-party archives (such as the Enterprise Vault extraction wizard) are slow and not geared up for performing automated mass exits. Once you’ve extracted files, you’ll need to make sure they are prepared properly for Microsoft. This includes the creation of a mapping file, so Microsoft knows what files(s) belong to who, and where you want them putting. Check out the steps you’ll need to carry out. Whilst it’s possible to automate the PST extraction and preparation process using third-party migration software, you’ll need to factor this additional cost in.
It can take a long time:You’ll need to allow 7-10 days for your data to be uploaded from the drives into Azure (as we said earlier, this is where your data is pre-staged) and then Microsoft offers an ingestion rate of 24GB per day. Using our 20TB example, this means your PSTs would take 860 total days to ingest.
It introduces an element of risk:When using multiple hops and manual interventions to move your data, there’s the potential for things to go wrong. Even though drive shipping uses Bitlocker encryption to protect your data in transit, there are many other steps that introduce the potential for human error, this includes the process of babysitting the extraction into PST files from your archive and the mapping of PST files to their owners. This, combined with the fact that extraction tools typically have no inbuilt error-checking, are unable to recover in the event of a failure, and no auditing, will make it difficult for you to prove chain-of-custody. Oh, and did I mention that PSTs as an interim file construct are prone to corruption?
Your source data needs to be static. If you’re migrating the contents of an email archive using drive shipping via PSTs you’ll ideally need to make your archive static during the course of the migration. This means stopping any archiving activity for the duration of your archive project, otherwise you’ll have the overhead of subsequently migrating any additions to your archive. We’ve encountered several projects where stopping archiving is not possible.
Shortcuts aren’t being addressed (and create confusion). You will need to have a game-plan for dealing with the shortcuts (also known as stubs) that typically link to archived items. Many enterprises end up migrating shortcuts along with regular emails into Exchange online mailboxes. Whilst in most cases it’s possible to retrieve the full item across the network from an on-premises archive whilst your migration is taking place, you’ll have various issues that emerge once your PSTs have been uploaded into Microsoft 365. This includes broken shortcuts (assuming at some point you will decommission your on-premises archive) and legacy shortcuts that can appear along with the full migrated item in the event of any eDiscovery exercise.
Other limitations:
- Message Size Limits of 150MB
- No more than 300 nested folders
- Doesn’t support Public Folders
- You don’t get flexibility on where your data is migrated to destination and split of data
- Volume restrictions of up to 10 TB
- A maximum of 10 Hard drives for a single import job

So should we use drive shipping for our migration?

In summary, the only time we can see drive shipping using PSTs as being beneficial is if you have:

Very slow network connectivity
Lots of inactive data to migrate. For example, archives belonging to ‘leavers’

Our email archive migration service uses a series of techniques to mitigate the impact of Microsoft throttling, enabling us to move archives directly from your archive into Exchange Online (either primary mailboxes or archives) at a rate in excess of 3TB a day. There’s also no overheads or time delays involved by extracting into PSTs first.

We can also schedule migration activity to coincide with less busy times on your network.

Also, the fact that we can move your data in one step, direct from source to target avoids the non-compliance risk of interim storage and human error.

We can also help you avoid moving everything. For example, by applying date ranges.

You can also avoid creating a storage overhead in the cloud by managing where data gets migrated to. I.e., by moving messages over a certain age into archive mailboxes or moving PSTs belonging to leavers into a separate (but indexed) Azure-based store.

On a final note, using interim PSTs is also an option when migrating journals from services such as Mimecast and Proofpoint, but there are a few things to watch out for when migrating into Microsoft 365. You can find out more about migrating journals to Microsoft 365 in this article.

Find out more for your PST migration project

Get in touch with our migration experts for an unbiased chat on the options open to you.

Get in touch

Sticky Bit 4: Migrating On-Prem Public Folders to Microsoft 365

Posted on 03/03/202121/07/2022 by Ross Moorhead

Essential has worked on some of the largest Public Folder migration projects in the world. Here’s a few tips from our gurus:

A few years back you didn’t have an option to migrate your legacy public folders to Microsoft 365 – in fact public folders on-premises were to be end-of-lifed. SharePoint was initially tabled as an alternative, but this didn’t ‘wash’ with a lot of Microsoft customers because it didn’t offer the same functionality and was over-complicated.

Microsoft quickly changed its position (no doubt following uproar from lots of disgruntled customers) and now you can take advantage of modern public folders – a service that seems to be hanging together reasonably well and growing bigger in capacity all the time. It’s now 100TB in total – it started out at 2.5 TB and then 50TB so it’s always worth checking here Exchange Online limits – Service Descriptions | Microsoft Docs!

As you might imagine, there are some caveats, clean-ups and other considerations that come into play if you want to make the move.

But first off, it’s worth getting a bit of background on the modern public folder construct vs traditional public folders:

The Modern Public Folder service is very different from the Public Folder database architecture in an on-premises environment. It basically uses regular mailboxes that are automatically linked together and load-balanced (for Microsoft 365) as your Public Folders grow in size. Being regular mailboxes they also benefit from being part of data availability groups (DAGs) instead of having to undergo painful public folder replication.

Here’s how the modern public folder to Microsoft 365 architecture works:

You kick off with a single, Primary Public Folder (PF) mailbox (which can grow up to 100GB in size)
Microsoft 365 detects when a PF mailbox is approaching the 100GB limit and uses an auto-split feature that creates a linked Secondary PF ‘overspill’ mailbox.
As the next mailbox fills up, another PF mailbox is added and content is automatically re-balanced across all the mailboxes.
This expansion continues until you hit an overall limit (at the time of the last update to this article it is 1,000 public folder mailboxes and 100TB in a single Microsoft 365 tenancy).
See this page for the latest info: https://technet.microsoft.com/library/exchange-online-limits(EXCHG.150).aspx
A PF hierarchy is maintained alongside the PF contents in the Primary mailbox.
This hierarchy is updated to reflect the new location of items as new PF mailboxes are added and as content gets ‘re-balanced’ across the available mailboxes.
Read-only copies of the PF hierarchy are also stored in each of the Secondary PF mailboxes and these are kept in sync with the Primary using Incremental Change Synchronisation (ICS).

The key thing to note that is that as far as users are concerned, although the Public Folder to Office 365 mail comprise multiple, ‘lashed together’ mailboxes, they can be viewed and navigated as a single, logical entity.

This is a really great PowerPoint by MVP Peter Schmidt that describes the whole thing in more detail:

https://www.slideshare.net/petsch/modern-public-folders

See also this Microsoft document for details: https://docs.microsoft.com/en-us/exchange/collaboration/public-folders/limits?view=exchserver-2019

Planning Your Migration

Can you migrate?

If you’ve already updated to using modern PFs on premises and are using Exchange Server 2013 CU15 or later, Microsoft has a public folder migration solution using batch migration scripts as described in this article:

https://docs.microsoft.com/en-us/exchange/collaboration/public-folders/migrate-to-exchange-online?view=exchserver-2019.

If you are using ‘old school’ PFs (aka legacy PFs) hosted on Exchange 2010 SP3 RU8, you can follow these instructions for batch migration.

https://docs.microsoft.com/en-us/exchange/collaboration-exo/public-folders/batch-migration-of-legacy-public-folders?view=exchserver-2019

Whichever version you’re migrating from you’ll need to run quite a few separate scripts (including a final synchronisation and switch which requires downtime) which means it can be quite complicated to use.

Do an Inventory and Have a Clean Up

Some of our customers store vital customer records in PFs. They also have a lot of rubbish in them and migration is a great opportunity to do a sort out.

Start by doing an inventory of your PFs at a ‘high-level’, and get statistics such as size, item count, owners, permissions and last accessed dates.

In order to make solid and defensible decisions around whether content can be deleted prior to migration you’ll need to do a LOT of deeper digging, however gathering initial meta-data can give you some excellent pointers. For example:

Removing empty and duplicate folders can be a quick fix.
Orphaned folders with an old last accessed date are a very obvious candidates for a clean up.
Knowing the owner of a PF (assuming it’s not ‘Administrator’) can help signpost who you need to contact in order to see if content can be disposed of.

As ever with records disposition decisions, seek to get the relevant data custodians to call the shots – don’t go it alone!

Bear in mind that a potential downside to deleting or excluding older/stale contents from your migration is that you could create an eDiscovery headache later. For example, an HR dispute may refer back to employment terms and conditions, pension fund arrangements, etc, that were published decades ago.

Analyse Your PFs for Potential Glitches

Given the inherent differences between the architecture of old PFs and Modern PFs, you’ll need to spend some time eliminating things that will upset the migration process. For example:

Check for stale permissions
Check there are no orphaned PF mail objects or duplicate PF objects in Active Directory
Check PF names – syntax errors in your legacy PF naming convention can cause problems. For example:
- If the name of a PF contains a backslash () it will end up in the parent PF when migration occurs.
- Trailing whitespaces within Mail enabled PFs and commas in the Alias field will also create synchronisation problems.
Check all mail enabled folders to see that they have the right proxy address.
If you have any forms, these need to be exported and re-imported into Office 365
If users have PF ‘favourites’, they will need to document these before you cut over, as they will disappear

Chunk Up Your Legacy Folders to Slot Nicely into the New Separate Mailboxes Model

As we said earlier in this article, Office 365 performs an auto-split and load-balancing function as PFs approach 100GB in size, but this process can take up to two weeks to complete. This is not usually a problem when you are populating a PF during ‘normal use’, but when you’re in a midst of a wholesale migration, you’ll be chucking data into Office 365 PFs at a rate of knots, and Office 365 can’t recalibrate itself fast enough.

Common to all migration approaches, therefore, is the need to take the Office 365 PF size restriction of 100GB per mailbox into consideration and effectively run scripts to ‘chunk up’ your PFs into separate PFs that are less than 100GB in size before you start your move. We suggest you check that your ‘chunks’ are split according to logical subfolders.

Don’t overlook that fact that some of the items in PFs may be archived, as this will not only impact how you do your migration, it will also impact your sizing analysis (as shortcuts to archived items can be a fraction of the actual item size). Check the message class to do this – e.g. IPM.NOTE.EnterpriseVault.Shortcut

There are many other considerations to take on board to ensure the best outcome post-move, such ensuring optimum retrieval times by putting PFs in a geographic location that’s near to users that will be accessing it. Ensuring the number of people accessing PFs is kept below 2,000 per mailbox is also recommended.

Post-move you’ll need to do lots of checking and you might also need to re-mail-enable mail-enabled PFs post migration as this attribute might not get migrated.

You can find other considerations here: https://technet.microsoft.com/en-us/library/dn957481(v=exchg.160).aspx

Find a third-party tool or a migration partner.

There’s a few tools around that can help you with your migration and the best ones start by enabling you to do an up-front analysis of the state of your PFs so you see (and address) what might ‘trip you up’.

The one we particularly like also has a neat algorithm that lets you map your current public folder hierarchy into new modern public folder arrangement.

Another area where you might need assistance is if you’ve been archiving public folders in the past, for example, using Enterprise Vault or EAS. At a push you can export archived content to PST files and use these as a mechanism for uploading on-premises PFs into Microsoft 365, but you need to know what you’re doing when it comes to splitting your PFs into ‘mailbox chunks’.

Get in touch with our team for a chat to see if we can help with your public folder migration (or at least point you in the right direction).

Let us migrate your Public Folders to Microsoft 365 (or elsewhere!)

We can simplify your Public Folder & Public Folder Archive migrations – or help you migrate to alternative platforms like Azure Get in touch to discuss your options.

Get in touch

Why there’re no good tools to migrate public folders to Microsoft 365

Posted on 21/06/201919/07/2022 by Essential Marketing

Along with PSTs, Public Folders are usually the last data repository to be dismantled when you’re migrating to Office 365. They’re typically big and full of static content, much of which is ‘ROT’ (redundant, outdated and trivial information).

Many public folders, however, contain active, business-critical information, especially where mail-enabled public folders have become enmeshed in business processes, such as the transport company we work with that uses public folders to manage and share all communications relating to all its shipments.

This means they can’t leave them behind. Apart from upsetting users and impacting efficiency, you might compromise future legal discovery situations.

For a while, Microsoft didn’t have the equivalent to public folders in Office 365. They eventually capitulated with the provision of a ‘modern public folder service’ in Exchange 2013, – even so, large ‘legacy’ public folders are proving to be tricky to move with any degree of ease and success.

Industry experts signposted Groups as the obvious alternative to public folders – and bet development dollars on tools to help move your public folders to Groups, but these never really got traction.

We are knee-deep in the migration of some of the UK’s largest public folder estates and discovering that even the best tools on the market are stopping short of providing a silver bullet.

In fact, many vendors, such as Quadrotech and MessageOps, have simply thrown in the towel when it comes to providing a fault-free public folder migration tool.

There’s no wonder!

Here’s a snapshot of just 6 obstacles we’ve been navigating recently in migrating public folders:

Large public folders

Part of the challenge of moving on-premises public folders to modern public folders is that you must partition them up into sub 100GB* sized chunks and distribute them in a way that ensures optimum accessibility and performance for users.

There’s a whole bunch of limitations you need to consider to ensure you stay within Microsoft’s recommendations. This includes a limit of 10,000 sub-folders and 1 million items per folder, the number of concurrent users accessing a public folder and many more – check out this article from Microsoft for more information.

Archived public folders

Archived public folders content complicates the sizing process. Emails that look like a few KBs in size when you run an initial analysis could be linked to much bigger emails and attachments (and cumulatively ‘blow’ the 100GB limit as you try to migrate). Taking a two-step approach and first re-hydrating archived content into Exchange is a good approach for knowing exactly what you’re dealing with, but it means you must be able to cope with an interim storage challenge and a protracted migration timeline.

**Public folders you think haven’t been archived**

Even if you feel confident that certain Public folders haven’t been archived, it’s highly likely they contain archived items that you don’t know about. This is because users might have dragged short-cutted (archived) items into them. To tackle this, you will first need to map all the relevant retrieval paths, which could be from several legacy archives, and then rehydrate the original items into Exchange prior to the migration.

Too many options as to where to move your public folders

Migrating legacy public folders to modern public folders in Office 365 is just one option open to you. Navigating your way through the alternatives (e.g. SharePoint, Groups, shared mailboxes, Teams – even non-Office 365 platforms) and understanding their limitations and benefits is mind-boggling, e.g.:

Groups – Good support for public folders collaboration features and better mobile support, but have a flat structure and no permissions granularity;
SharePoint – great versioning support, check-in/check-out, but not mail-enabled.
Shared Mailboxes – Good hierarchy support and granular permissions, but not possible to mail-enable specific folders and customize the mailbox view to exclude default folders such as calendar, contacts, drafts, deleted Items, sent items, etc.

You also have the option to archive off static public folders content to other locations such as a cloud-based archive, in which case you need to consider things like end-user access, access control, eDiscovery and so on.

Migrating public folder permissions

When you relocate public folders content, mapping access rights correctly is vital – especially with concerns like the GDPR to contend with. This step is relatively straightforward if you’re migrating to public folders online, but not if you’re moving to a non-Office 365 platform, such as an Azure-based archive.

Even though you can extract email recipient information as the basis of governing access, coping with very large public folders combined with the additional overhead of expanding the members of distribution lists, creates a whole new challenge. You might also take your migration as an opportunity to tidy up and streamline permissions.

Tackling invalid characters in public folders

Trailing spaces, the wrong sort or dash (that longer dash they use a lot in the US), back and forward slash – these all need to be removed from your legacy folder names prior to migration. Otherwise, the default may to move their contents to the parent folder and blow the storage limit. In actual fact, this is the easiest bit to tackle.

Final thoughts

Each project has its unique technical challenges. It’s clear that the key to working out your best migration strategy is to perform in-depth analysis before you start.

Check out our free public folder analysis tool.

It’s unlikely you’ll find a single silver bullet for your specific needs, but following your analysis we’ll be happy to make recommendations and share with you what we’ve learned so far in terms of the best tools and techniques around to help with your move.

Having difficulties migrating public folders to Office 365?

Analysis is key! Request a quote and get a free trial to our analysis tool

Get a Quote