Demystifying Commvault file stubbing, a tale from Sayonara to Hakuna Matata.

How stopped worrying and loved Commvault File System Archiving

Recently, I came across a situation where there was an issue with a legacy Windows Server with several Terabytes of stubbed files in Commvault’s archive storage.  When the time came to migrate this data to a new server, they kicked off a RoboCopy to transfer the data whilst the Commvault Services were disabled to prevent triggering the archive recall.  It was observed that the RoboCopy block-level copy transferred the Data with the ACL’s and Attributes; but Commvault could recall the data from the new server.  Clearly frustrated, there was misplaced blame that was attributed to Commvault and not on the original decision to use RoboCopy.  But wait, why blame RoboCopy if it copies the File Attributes and Commvault’s stubbed files are tagged with Offline and Link attributes?

To understand this, it helps to have a stickybeak into the NTFS architecture.  The NTFS File system contain ‘File Streams’ the data written and additional information about the file attributes and properties.

Additional capabilities have been incorporated to the NTFS over the years, and the ‘Reparse Point’ File Stream was introduced way back in Windows 2000’s NTFS v3.0 file system.  Since then, NTFS files contain reparse tags and reparse data that are intercepted by FileSystem (mini) Filter drivers.  Due to the way the I/O is intercepted, the Mini-filter drivers are executed in kernel-mode and any user-mode software (e.g. RoboCopy) that reads the files that contain a Reparse Point data stream will first attempt to execute a recall of the stub via the logic build into the filter driver before it can access the file. A file that has been stubbed by Commvault file legacy archiving (GxHSM, or more recently ‘cvmhsm’) will be tagged with ‘Hierarchical Storage Management’ data within the Reparse Point File Stream.  You can see which of the minifilter you have installed (in this case CVDLP) by running fltmc filters.  Note that if you cannot load either the gxhsm or cvmhsx driver, then you must use the Commvault Edge Plug-in for Windows Explorer.

PS C:\> fltmc filters
Filter Name                     Num Instances    Altitude    Frame
------------------------------  -------------  ------------  -----
bindflt                                 0       409800         0
vsepflt                                 0       328200         0
WdFilter                                5       328010         0
storqosflt                              0       244000         0
wcifs                                   0       189900         0
cvmhsm                                  5       181400         0
CldFlt                                  0       180451         0
CVDLP                                  10       145180         0
FileCrypt                               0       141100         0
luafv                                   1       135000         0
npsvctrig                               1        46000         0
Wof                                     1        40700         0

What this table shows is that each filter instance will be loaded in descending Altitude order.  Altitudes are categorised into various Load Order Groups such as ‘FSFilter Anti-Virus’ which are loaded before the ‘FSFilter HSM’ group.

Filter Manager Concepts – Windows drivers | Microsoft Learn

There are a number of ways the fsutil tool can be used (fsutil reparsepoint, fsutil sparse) to inspect the stub information, but I like the fsutil file layout /v show the full file layout information. 

Here we can see the Sparse, Offline, and REPARSE_POINT stream contains the Reparse Tag Value and GUID that has been assigned to Commvault by Microsoft.  Incidentally, if you have access to the Windows Driver Kit, the Reparse Tag Value and GUID shown above are in the ntifs.h header file.  The hex stream above contains all the Remote Storage information required by the driver (e.g. associated JobId 1225 and Storage Policy Id 4) that the Client and the CommServe need to recall the stub file.

If the Commvault “Driver for File Archiver” was not loaded then RoboCopy will hang with “The file cannot be accessed by the system” until RoboCopy reaches your chosen retry limit, or until the Commvault Services are running and the Commvault “Driver for File Archiver” has been installed.  That is unless when you configure the RoboCopy to limit the number of retries in order to avoid delaying your migration.

But they had the driver installed, so why did RoboCopy not copy the stubs?

There are a number of requirements if don’t want to trigger an in-place recall before migrating the stubs. Such as,

• Commvault Services must be running on the source.
• Commvault File Archiver Driver must be installed (requires reboot).
• Use Commvault’s preferred method of copying stubs GxHSMUtility.
• Or the non-supported ‘RoboCopy /B’ with the Backup Switch that will copy the reparse point information.
  
• The Job Id must be accessible (not partially copied) in an online Storage Policy Copy.
• The Anti-Virus Exclusions have been applied because the ‘FSFilter Anti-Virus’ group is loaded prior to the ‘FSFilter HSM’ minifilter that will trigger the recall.
• The Stubs should not have undergone an offline relocation into another folder. If you have then you will likely notice that the next archive job which will not update the 0kB stub reparsepoint metadata and that will result in a failed stub recall.  Note: This was the root cause for my customer.

• If you discover that you have moved the archived folders to another mapped drive or folder, then you can look at the reparsepoint metadata and trial copying the stubs back to the file original archive folder and then retry the stub recall. 
• If the files were moved a long time ago, then it is possible that the Archiving job has been aged out of the online Storage Policy Copies, and you would need to recall cloud/tapes that hold the backups.  The reparsepoint metadata includes the Job Id and its Epoch Time Stamp that may help you identify the cloud/tapes that will be needed, otherwise you can contact Media Management Team at Commvault Support.
• If the stubs were incorrectly migrated (0kB without reparsepoint metadata) and the source client is unusable, then you can use the Commvault UI to recover the data from previous archive jobs.  Sometimes there is no choice, but I am fond of this as bulk-migration method. Client-side recalls are far more elegant since each recall is a single operation as it also functions as the Single source of truth.
  When you use the Console UI to trigger recovery operations, you should factor in the likelihood that your subclient content rules could have been updated over time, and this can make the process cumbersome if you need to go back in time to browse for past stubbed files/folders that were moved incorrectly.

Seems like a complicated problem, is it unique to Commvault?

Speaking from personal experience, there are common ways you can encounter a similar problem.  Popular third-party Cloud Storage products like DropBox and even Microsoft’s OneDrive which allow you to extend your file system employ a very similar stubbing technology.

A while back my laptop had been experiencing a number of BSOD’s which I would later discover that it was the result of an erratic Wi-Fi Extender, but to my frustration I discovered that my user profile corrupted.  Whilst thankfully my road to recovery was at least simplified because my files were still accessible in Safe Mode, my System Restore points were useless because the ‘CldFlt’ minifilter for OneDrive doesn’t get loaded during the System Restore.  WTH!  Even MS-DOS would have prompted me with “General failure reading OneDrive. Abort, Retry, Ignore, Fail?” (sarcasm), but it would have been really useful in this case to allow me to skip my OneDrive files which were in the cloud anyway.  In the end, it was just easier to create a new profile and double check that my laptop files were still in my backups.