PST file migration

Is drive shipping best for large PST migrations to Microsoft 365?

Posted on 25/04/202222/07/2022 by Alethia Erchen

Do you need to migrate thousands of PST files, really big mailboxes or large email archives to Microsoft 365 (and is drive shipping the answer?)

What is a PST file?

A PST file is an Outlook Data File (.pst) that can be used to store messages and other Outlook items. From the ‘outside’ they look like a single file, but inside they have the structure of a pseudo mailbox.

Larger organisations in particular can have a legacy of many thousands of PST files sitting on individuals’ local drives, home drives and file servers. They would historically get created as a way of staying within mailbox quota (when Exchange mailbox quotas on premises where perhaps not as generous as they are online). PST files were also used as a convenient way of preserving mailbox contents – perhaps after an individual had left the company. In fact PSTs used in these scenarios are widely considered a pest – especially when organisations switch to the cloud.

PST files can also be used as an interim store when migrating between platforms, as many mail systems can export emails and attachments into PST format, and conversely, import content from a PST file. It is this ‘use-case’ that is the subject of this particular article…..

What is drive shipping?

Microsoft’s drive shipping service is multi-step process via which large amounts of data can be uploaded into Microsoft cloud via an interim device which is physically shipped to a Microsoft location.

This is how it works:

Instead of transferring data across the network, the technique involves writing PST files to a hard drive along with a mapping file (for example, migrate PST <filename> to <username> primary or archive mailbox).

The (encrypted) hard drive is then physically shipped to a designated Microsoft location from where data centre personnel pre-stage the contents into Azure. The files are then ingested into Exchange Online according to the supplied mappings.

If you have TBs of emails you want to move into Microsoft 365, drive shipping via PSTs is an option open to you.

This Microsoft article goes into all the different iterations of what you need to do and the cost of using this service.

The same technique can be used to perform migration of mailboxes that are over 100GB in size or if the user’s mailbox contains one or more messages that exceed the 150-megabyte (MB) message limit (in which case resorting to PST files is recommended).

Using interim PSTs is also an option for migrating the contents from large email archives such as Enterprise Vault and EMC SourceOne, or email journals form platforms such as Mimecast.

The question is: Is drive shipping PSTs a good option for your large email archive migration?

Here are the pros and cons:

Pros

It’s low cost…on paper*. The cost to import PST files to Microsoft 365 mailboxes using drive shipping is $2 USD per GB of data.
It minimises impact on your network: if you have a sub-optimal network that cannot handle large amounts of data being transferred, using a data drive to ship PSTs to Microsoft negates any network concern. Even on networks that can support around 500Mbs you can experience slow performance when you start to drive a large migrations alongside regular user activity.
It avoids the impact of Microsoft throttling:Microsoft applies throttling to avoid overloading its servers. Although you won’t experience the effect of throttling when using native Microsoft mailbox moves to migrate your mailboxes, many email archive migration solutions use the EWS protocol to move your data, and this protocol is subject to throttling, although Microsoft has made throttling easy to ease off during the course of a bulk migration.

Cons

*It can work out expensive:At face value, $2 USD per GB is cost-effective, for example, a 20TB project would be $40,960 to ‘drive ship’, but this does not include the added overheads of getting your data onto the drives (see next point).
PST preparation is labour-intensive. Suffice to say that manually extracting data from archives into PST files and then preparing them for upload can be super time-consuming. Native tools for extraction out of third-party archives (such as the Enterprise Vault extraction wizard) are slow and not geared up for performing automated mass exits. Once you’ve extracted files, you’ll need to make sure they are prepared properly for Microsoft. This includes the creation of a mapping file, so Microsoft knows what files(s) belong to who, and where you want them putting. Check out the steps you’ll need to carry out. Whilst it’s possible to automate the PST extraction and preparation process using third-party migration software, you’ll need to factor this additional cost in.
It can take a long time:You’ll need to allow 7-10 days for your data to be uploaded from the drives into Azure (as we said earlier, this is where your data is pre-staged) and then Microsoft offers an ingestion rate of 24GB per day. Using our 20TB example, this means your PSTs would take 860 total days to ingest.
It introduces an element of risk:When using multiple hops and manual interventions to move your data, there’s the potential for things to go wrong. Even though drive shipping uses Bitlocker encryption to protect your data in transit, there are many other steps that introduce the potential for human error, this includes the process of babysitting the extraction into PST files from your archive and the mapping of PST files to their owners. This, combined with the fact that extraction tools typically have no inbuilt error-checking, are unable to recover in the event of a failure, and no auditing, will make it difficult for you to prove chain-of-custody. Oh, and did I mention that PSTs as an interim file construct are prone to corruption?
Your source data needs to be static. If you’re migrating the contents of an email archive using drive shipping via PSTs you’ll ideally need to make your archive static during the course of the migration. This means stopping any archiving activity for the duration of your archive project, otherwise you’ll have the overhead of subsequently migrating any additions to your archive. We’ve encountered several projects where stopping archiving is not possible.
Shortcuts aren’t being addressed (and create confusion). You will need to have a game-plan for dealing with the shortcuts (also known as stubs) that typically link to archived items. Many enterprises end up migrating shortcuts along with regular emails into Exchange online mailboxes. Whilst in most cases it’s possible to retrieve the full item across the network from an on-premises archive whilst your migration is taking place, you’ll have various issues that emerge once your PSTs have been uploaded into Microsoft 365. This includes broken shortcuts (assuming at some point you will decommission your on-premises archive) and legacy shortcuts that can appear along with the full migrated item in the event of any eDiscovery exercise.
Other limitations:
- Message Size Limits of 150MB
- No more than 300 nested folders
- Doesn’t support Public Folders
- You don’t get flexibility on where your data is migrated to destination and split of data
- Volume restrictions of up to 10 TB
- A maximum of 10 Hard drives for a single import job

So should we use drive shipping for our migration?

In summary, the only time we can see drive shipping using PSTs as being beneficial is if you have:

Very slow network connectivity
Lots of inactive data to migrate. For example, archives belonging to ‘leavers’

Our email archive migration service uses a series of techniques to mitigate the impact of Microsoft throttling, enabling us to move archives directly from your archive into Exchange Online (either primary mailboxes or archives) at a rate in excess of 3TB a day. There’s also no overheads or time delays involved by extracting into PSTs first.

We can also schedule migration activity to coincide with less busy times on your network.

Also, the fact that we can move your data in one step, direct from source to target avoids the non-compliance risk of interim storage and human error.

We can also help you avoid moving everything. For example, by applying date ranges.

You can also avoid creating a storage overhead in the cloud by managing where data gets migrated to. I.e., by moving messages over a certain age into archive mailboxes or moving PSTs belonging to leavers into a separate (but indexed) Azure-based store.

On a final note, using interim PSTs is also an option when migrating journals from services such as Mimecast and Proofpoint, but there are a few things to watch out for when migrating into Microsoft 365. You can find out more about migrating journals to Microsoft 365 in this article.

Find out more for your PST migration project

Get in touch with our migration experts for an unbiased chat on the options open to you.

Get in touch

Top 6 PST Elimination Challenges when Migrating

Posted on 18/09/201411/02/2021 by Essential Marketing

Warning:

.psts are like vermin.

Experts agree they are a pest and IT departments across the world are all too familiar with the support nightmares they create. They also carry the risk of data loss & exposure.

Apart from being massively out-of-control in most organisations (difficult to locate on users’ hard disks and containing unknown evils), they are inherently difficult to grapple with.

Having worked on PST elimination projects for over fifteen years now, we have encountered quite a few challenges and gotchas. Here’s just a few to bear in mind if you are planning to get rid of your PSTs.

1. Users Get Attached to their PSTs

Having been forced to use PSTs for so many years to stay within enforced quotas, end users have become used to them ‘being there’. We know of law firms whose partners have created PSTs for each major client, and store them on their laptops. Removing PST files without explaining why; and without creating a familiar and easy way to access the contents of PSTs post migration, will be very dimly received.

2. PSTs are only accessible by one user/application at a time

PSTs are only really designed to be accessed by one user or application at a time. This creates a challenge where PSTs are opened when a user logs into Outlook. If this happens as a matter of course at your organisation you will need some way to ensure users don’t access their PSTs during your migration project (see also next point). You will also have to work out how you are going to tackle users with laptops that only connect to the network intermittently, and users that always have their PSTs open.

3. PSTs are usually an ‘all or nothing’ thing

If you look at a PST in a directory listing – it’s just one file. It’s effectively a container file, and with most solutions that you can use to move or eliminate PST contents, you’re typically looking at taking the whole file with you. This can mean taking a whole bunch of stuff you just don’t need or that falls outside of your retention policy. Having said this, it’s possible to take a more granular approach and just select the items you want to move according to different criteria. We have found this saves a lot of time and network bandwidth (as well as archive storage).

4. PSTs might be changing all the time

If you plan to tackle PSTs we recommend you create an Outlook Group Policy that makes the existing PSTs read only and prevents new PSTs from being created. This will put your PSTs in a ‘stable’ situation while you are tackling them.

5. Be Careful Assigning Ownership

From a legal and audit point of view, who owns a PST file can be vital information. Where PST files are stored on network shares it might seem to be obvious who the owner is, but this depends on how structured the shares are and permissions on each folder within the share. Where “team” folders exist, each PST file could contain messages which are “owned” by several people. Where PST files are stored on desktops then you need to examine what happens when one user leaves and another joins. Are the desktops completely rebuilt or just new profiles configured? In the latter case PST files could be lurking that belong to users who have left the organisation many moons ago!

6. What happens post PST removal?

Users who are working offline will tend to have PST on laptops. Using Outlook in cache mode (which creates an OST file) is the logical alternative for enabling offline access, so following a PST migration exercise you need to make allowance for the extra network traffic that will be created during the first offline synch after migrating them.

There are many more PST tips we have

Get in touch and find out more about our PST migration services.

Get in touch

Personal devices and social activity in your company

Posted on 28/02/201228/12/2020 by Essential Marketing

There’s been a lot of blog posts flying around recently around the growing trend towards ‘Bring your own device’ (BYOD). Deloitte and Citrix are some of the big names to announce recently that they are rolling out BYOD internally.

Given that CDH research confirms 1 in 10 employees already use their own devices to access work email, and other surveys cite even higher figures, is this trend an inevitability? Systems Professional (HP’s go-to partner for VDI solutions) tell us they are already seeing an increase in businesses looking to deploy VDI in part to help manage increased usage of user-owned devices.

With almost a third of the execs polled in Deloitte’s recent survey believing that there are likely ‘rogue’ devices connected to the network- how do they intend to know for sure?

Organisations planning a BYOD approach will need to consider how this will be managed in practice. Deep-dive monitoring solutions like Mailscape will certainly help businesses monitor which devices are connected- and the status of the device right down to the OS, and some sort of solution to monitor connectivity and policies should be considered from the outset.

Citrix certainly seems to believe BYOD and desktop virtualisation is an inevitability. Having said all of that, and despite having read numerous articles on the subject of late- we have lost count of the amount of trends which turn out to be more hype than reality. We’ll keep an open mind for the time being…

I’ve often posted a status on Facebook and thought nothing of it, but the new changes made me think twice about my profile on there and the prospect of leaving a trail of my personal life brought me to the decision to delete my account, but it also made me think about what I may also be leaving in the archives of my current and past employers(s) that will be virtually impossible for me to delete.

Some organisations routinely keep archives of all emails sent and received via company systems, all instant messages, any interaction on social networking sites, files and so on. I know, because we provide the technology to do this.

Even in the last few weeks new software has arrived from Microsoft (PST Capture) that allows organizations to grab personal archives (PST files) containing copies of emails that were previously under the control of the individual. Once captured into the central Exchange store, the contents of PSTs can be searched by the company, deleted by the company and potentially retained for an eternity.

It depends on your train of thought of course, the minute you save something on to a work sever or PC is it still your data or is it now the responsibility of your company?

“Let’s keep it anyway – just to be safe”

Posted on 21/02/201211/02/2021 by Essential Marketing

Even though Microsoft offers free tools to import your PST files into Microsoft 365, does anyone really relish the thought of migrating ALL of their PST files into Exchange?

Probably not, because:

You end up moving old rubbish
It will pound your network
Users might not like it

Having worked with hundreds of customers to roll-out email archives, PST migration is always the painful final phase that often gets brushed under the carpet for the aforementioned reasons.

The over-arching requirements for anywhere, any device data accessibility and centralised data governance, however, may be the final nudge to get your PST migration project management backing.

So is it acceptable to filter out the rubbish before you siphon it into your shiny new Exchange environment?

At the very least it’s surely prudent to get a policy agreed that will define your PST migration including whether to migrate:

Content that outdates your email retention policy
Folders marked ‘Personal’
Content that pre-dates your switch on of Journal capture
Duplicate PSTs

A PST migration project means bracing yourself for decisions about what to migrate and no-one likes decisions! But we like the advice from Craig Ball, a Texan trial attorney and certified computer forensic examiner:

‘To Preserve broadly is safe, but expensive. To Preserve carefully is safe and cost-effective”.

The truth is, any enterprise-level PST migration exercise – even if you gather up everything – is not easy and you may struggle with free Microsoft tools.

PST Migration Challenges

Check out the top 6 PST Migration Challenges

Learn more

Fallacy: An archive does not allow you to permanently delete an email

Posted on 02/02/201211/02/2021 by Essential Marketing

We read with interest a recent Mimecast blog which refers to a TechCrunch article on the News Corporation phone hacking scandal. The archiving vendor alludes to the concept of an email archive making it possible to permanently delete an email from circulation.

Perhaps we misread the intention of the writer, but here’s a home-truth speaking from experience:

Even if you maintain a central archive of your emails, be it on-premise or the cloud, deleting any given email from said archive is highly unlikely to delete all copies of the email.

This is because any given email is likely to exist in many other places besides an archive (and print outs in a crate), including:

on past backups of your email and archive servers
in personal email archives (aka PST files) that can exist on the user’s own hard disk or even a memory stick, and
assuming it wasn’t just an internal email, in the archive of the organisation with whom an incriminating email was exchanged.

The first 2 examples are becoming easier to tackle. Earlier this month Microsoft released its own tool to find and ’round up’ the contents of PST files and for many years now our UK-based email management organization has been delivering services and software to aid the search and recovery of emails from backups, PSTs and other locations.

So yes – implementing a robust archive along with sound and defensible policies for deletion can significantly reduce your eDiscovery costs and limit your exposure, but if the crunch comes, the incriminating emails can and will come out of the woodwork more easily than you think.