Backup Exec Deduplication Verify, Restore and Duplicate to Tape Performance Tuning

book

Article ID: 100027502

calendar_today

Updated On:

Cause

In order to read out the contents of a Deduplication Storage Folder, all of the data must be reassembled into a data stream and provided to Backup Exec.  There is no performance gain from being deduplicated when reading the data out.  Unlike the backup, all the data must be transferred when it is being read from the folder.

There are however some settings that can be experimented with to improve the read performance of a Deduplication Storage Folder.

Resolution

Backup Exec Deduplication Verify, Restore and Duplicate to Tape Performance Tuning

This document provides a list of tunable parameters that can affect the performance of a Deduplication Storage folder.  Please note that modifying these setting is not a guarantee that the change will improve performance.  The performance of a Windows system is affected by a complex mesh of parameters, and the only way to know for sure how these settings will affect the overall performance of the Deduplication Storage folder is to try them.

Terminology:

Deduplication Storage Folder Host – The Windows system where the Deduplication Storage Folder is located (attached).  In most cases, this is the Media Server hosting the Deduplication Storage Folder.

DSF – Deduplication Storage Folder

Deduplication Storage Folder Settings

Edit the \etc\puredisk\contentrouter.cfg file on the DSF Host and set the ReadBufferSize to 1048576.  Also change the PrefetchThreadNum from 1 to 4 or higher.  The DSF services must be restarted before these settings take effect.

Patching the Windows OS – Make sure to install at least SP1 on Windows 2008 R2

In Windows 2008 Server R2 without Service Pack 1:

- Under heavy I/O the kernel may not perform as expected. The cause of this issue is documented in Microsoft KB article 982383, which says that a computer with heavy I/O may encounter decreased performance.  To correct this issue, apply Service Pack 1 (SP1) for Windows Server 2008 R2. If SP1 cannot be applied, then apply the Microsoft Hotfix documented in Microsoft KB article 982383.

- Microsoft released a pre-SP1 patch for TCP loopback latency issues (around January 2011) which can be found here: https://support.microsoft.com/kb/979612

Windows Settings

- Windows 'Power Options' settings in Windows: - By default, Windows sets the Power Options to Balanced, which attempts to balance the power consumption of the hardware, but may sacrifice performance. Performance improvements have been found when the Power Options setting is set to 'High performance' or 'Performance'. These settings can be found in Windows Control Panel, Control Panel\All Control Panel Items\Power Options.

Network Settings

Disable (turn off) all checksum offload settings on the NICs on the DSF (Deduplication Storage Folder) Host.  Reference - https://www.veritas.com/business/support/index?page=content&id=000034042

TCP Chimney Offload and Recieve Side Scaling (RSS) settings of the network interface: - In previous articles, such as 294308 and 29008 it was recommend to disable TCP Chimney Offload. For Windows 2008 R2 platforms, performance gains have been realized when TCP Chimney Offload is enabled. Microsoft article 912222 discusses TCP Chimney Offload and RSS and the purpose of each. Here is a summary from Microsoft article 912222: "The architectural goal of the Scalable Networking initiative is to scale applications in the Windows family of operating systems to new levels of performance at a variety of cost points. Because of the variety of cost points and because of the variation in workload, the initiative includes the following two fundamental approaches: Stateful offloads Stateless offloads TCP Chimney is a stateful offload. TCP Chimney offload enables TCP/IP processing to be offloaded to network adapters that can handle the TCP/IP processing in hardware. Each TCP Chimney offload-capable network adapter has a finite number of connections that it can support in hardware. TCP connections will be offloaded to hardware as long as the hardware can support these connections. After the hardware offload connection limit has been reached, all additional connections are handled by the host stack. RSS and NetDMA are stateless offloads. Where multiple CPUs reside in a single computer, the Windows networking stack limits "receive" protocol processing to a single CPU. RSS resolves this issue by enabling the packets that are received from a network adapter to be balanced across multiple CPUs." Microsoft has a brief presentation available on TCP Chimney Offload and RSS available at MSDN. The article is called "High-Performance Networking With NDIS 6.0, TCP Chimney Offload, and RSS". The presentation discusses these options in context of a 10GbE network. Other types of interfaces may not have these options available, but they are worth looking into.

The bottom line is that if the NIC driver implements the TCP Chimney stack well, it can improve performance.  If it is not, it can harm performance and connectivity.

Update Software and Firmware

Make sure that the latest firmware and drivers are installed for server hardware (HBAs, NICs, etc).

Disk Performance

Disk I/O performance: - Care should be taken when creating LUNs, VDISKs, disk groups and volumes. Once created, disk I/O testing should be performed to prove read/write of 200MB/sec or better using for example Microsoft's 'sqlio' tool for Windows. Then and only then should the Deduplication Option be installed/configured. - Use a 64K allocation unit size when formatting NTFS for the DSF volume. The default is 4096 bytes. 64K is recommended for better performance. – 

Disable Antivirus checking of the NTFS volume for the DSF.  Not only will this improve performance, but can prevent corruption of the DSF by the anti-Virus program if it decides to quarantine, or delete a DSF data file.

Windows Indexing Service for the NTFS volume for the DSF. Veritas highly recommends administrators right-click each NTFS volume drive letter used by the DSF, click Properties and in the General tab, deselect the 'Allow Indexing Service to index this disk for fast file searching' checkbox.

Consider disabling 8dot3 name for the NTFS volumes.  Note that this is at best a minor performance improvement.  To do it, use the following command.

fsutil behavior set disable8dot3 D:   (Where D: is the drive letter for the volume where the DSF is located.

Issue/Introduction

Poor read performance (also referred to as rehydration) from a Deduplication Storage Folder results in slow Verify and Restore operations