Domino attachment and object service

Domino attachment and object service = DAOS.

With the release of version 8.5, IBM Lotus Domino server employs the Domino attachment and object service to save significant space at the file level by sharing data identified as identical between databases (applications) on the same server. Document attachments are the first components to use the DAOS feature in Lotus Domino.
In databases that use DAOS, Lotus Domino no longer saves a separate and complete copy of every document attachment. Instead, the server saves a reference to each attached file in an internal repository, and it refers to the same file from multiple documents in one or more databases on the same server. When an attached file is large and a message containing it is broadcast to thousands of users, creating a separate copy of the message for each recipient could require several gigabytes of disk space. Multiple copies of the same attachment often also proliferated in mail threads with multiple replies. With DAOS enabled, disk space usage is substantially reduced.

Figure 1. DAOS architecture

Planning a Lotus Domino attachment object store
Use of an attachment object store is optional, and it involves considerable planning before you can implement it in Lotus Domino.
You can mark databases on a Lotus Domino server for participation in attachment consolidation by enabling consolidation on the DAOS tab in the Server document, and by ensuring that every database that you want to include in consolidation has the "Use Domino Attachment and Object Service" advanced database property selected. DAOS also requires transaction logging to be enabled. DAOS stores a single copy of each attachment in a central mapped repository. After you enable attachment consolidation on a server, all databases on the server that are included in consolidation use the repository to store attachments.
When attachment consolidation is enabled and a user saves an attachment, the body stored in the document contains a reference, sometimes called a ticket, to the attachment, which identifies the attachment in the repository. Consolidation occurs immediately; you do not have to wait for a task to run before disk space savings occur in the size of documents with attachments.

Back to top

Attachment consolidation threshold, object store base path and retention period for DAOS unreferenced objects
Administrators can use the DAOS tab in the server document to specify a minimum size that an attachment must reach before being consolidated and to specify the parent directory on the server for the repository.
Attachment consolidation is completely transparent to users. When a recipient opens a document, the attachment icons display the same whether the document's database is on a server where consolidation is enabled or a server where it is not. Users can delete, save locally, replace, and perform any other familiar tasks with attachments. If users delete or replace an attachment, or if they create new documents or messages containing copies of existing attachments, the server adjusts the references to each attachment in the repository as necessary, using mapping to determine which attachments on the server are identical.
Before you set up attachment consolidation, decide where to place files and how large files should be before consolidation. On each server that uses attachment consolidation, you should specify the directory where you want to create the resulting repository of shared file attachments. If the number of DAOS (.NLO) files in the directory becomes too large for your operating system limit, Lotus Domino creates additional subdirectories as needed.
The attachment repository, by default, is created relative to the logical directory structure that is controlled by the server. You can create the directory anywhere on the server or a connected file drive by specifying a full path.

Back to top

Attachment consolidation for mail files
When you use attachment consolidation for mail files, also use it for any MAIL.BOX files on the server.
When you calculate the size of a mail file to determine whether it conforms to the configured mail quota or the warning threshold limits, Lotus Domino treats attachments stored using DAOS as though each user owned the entirety of the attachment file. Thus, the full size of every message delivered to a mail file that uses DAOS counts against the mail file quota. Likewise, when a user deletes a message that is linked to a DAOS-enabled database, the full size of the message is removed from the mail file quota.
NOTE: The actual file size of the mail database that uses attachment consolidation need not necessarily reflect its logical size.

Back to top

On-disk structure
Consolidation works for all databases that are on-disk structure ODS 50 and later. Databases with an earlier ODS do not include the DAOS setting in their advanced properties, but you can pre-enable them for consolidation in the Lotus Domino Administrator client so that they are included in consolidation when later upgraded to ODS 50.
To upgrade all newly created databases to ODS 50 so that you can mark them for attachment consolidation, enter the following setting in your NOTES.INI file:
Create_R85_Databases=1
Attachment consolidation is supported on Windows®, Linux® SLES 10, AIX®, and System i™.
Attachment consolidation is not limited to mail; it occurs as soon as an attachment is saved in any document of any database on the server where the feature is enabled.
NOTE: Attachment consolidation is not supported for DB2®-enabled Lotus Notes databases.

Back to top

Compact options
In addition to enabling the Lotus Domino attachment and object service property to consolidate new attachments in a database, you can use the Compact option detailed in table 1 to move attachments in existing documents into the DAOS repository while performing a copy-style compact. This option is not available for in-place compaction.

Table 1. The Compact option

Option	Command-line equivalent	Description
Use Lotus Domino Attachment and Object Service	-c -daos on|off	When it is set as on, this option moves attachments from existing documents to the DAOS repository and then compacts the database.

Back to top

How attachment consolidation works
The following list describes what happens to an attachment during consolidation.

• An administrator enables the feature for an entire IBM Lotus Domino server on the DAOS tab in the Server document.
• A database administrator, administrator, or full access administrator sets the "Use Domino Attachment and Object Service" database property to include one or more existing databases in consolidation, or any user creates a new database from a template whose DAOS property is enabled and ready to be inherited.
• In any DAOS-enabled database, if a user saves a document containing one or more file attachments larger than the size specified on the DAOS tab, the Lotus Domino server stores a reference to the file attachment in the object header of the document, and it stores a single copy of the file attachment in the attachment repository on the server.
• When any user opens the document again, the header activates a link to the stored file attachment. The file attachment displays as if the file was actually stored in the document.
• If a user edits the document and saves a new version of the attachment, Lotus Domino adds the revised copy of the file attachment to the attachment repository, deletes the reference in the document to the original file attachment, and creates a new reference to the revised attachment.
• If users later create multiple documents (such as replies in an email thread) containing copies of the same file attachment, the Lotus Domino server places a reference in every document to the single copy of the file attachment in the repository.
• If a user later deletes the attachment, Lotus Domino deletes only the reference in the document header. The file attachment in the repository is not affected as long as it is still used by other documents.

If all users delete the attachment from all documents to which it had been attached, Lotus Domino removes the file from the attachment repository. The attachment is removed immediately unless the administrator has specified a number of days for deferred deletion.

Back to top

Setting up and enabling DAOS
To specify server settings for attachment consolidation, follow these steps:

1. From the Lotus Domino Administrator, click the Configuration tab and then expand the Server section.
2. Select the server document to be edited, and then click Edit Server.
3. Click the DAOS tab.
4. Enable or disable the use of attachment consolidation by completing the fields shown in table 2.

Table 2. Attachment consolidation details

Field name	Enter
Store file attachments in	Choose one: DAOS. The server consolidates attachments saved in any document in any database included in consolidation. A database is included when the database property "Use Domino Attachment and Object Service" is selected. NSF. The server stops using attachment consolidation, and any new attachments in participating databases are stored in documents.

5. Enter the information shown in table 3, and then click Save and Close.

Table 3. Attachment properties

Field name	Enter
Minimum size of object before Lotus Domino stores it in DAOS	The minimum total size, in bytes, of any attachment to be consolidated (the default is 4096 bytes). Enter a number. Consolidating attachments of very small size is not worth the processing effort because it does not yield much disk space saving.
DAOS base path	The relative or full path to the attachment repository. For example, if you enter DAOS The DAOS base path is placed in the Lotus Domino data directory as follows: Example on Windows: C:\LOTUS\DOMINO\DATA\DAOS Example on Linux (SLES 10) or System i: /local/notesdata/DAOS If you want the directory elsewhere on the server, enter a full path, for example, on Windows: C:\DAOS If the directory you specify does not exist, Lotus Domino creates it for you. NOTE: Make sure that the Lotus Domino administrative user has file-access permissions in the directory appropriate to the server's operating system.
Defer object deletion for n days	Use this setting to keep objects in DAOS from being deleted for an additional number of days after the last reference to them from any document is deleted. Enter a number from 0 to 9999. This setting is useful if an unreferenced DAOS object is mistakenly deleted and needs to be restored. You can delete all unreferenced objects explicitly at any time by using the DAOS Prune command.

6. To put the new configuration into effect, restart the server.

Figure 2 shows the DAOS settings.

Figure 2. DAOS settings

To mark a selected database for attachment consolidation, follow these steps:

1. From the Lotus Domino Administrator, choose File - Application - Properties.
2. Click the Advanced tab. (The icon on this tab is a beanie.)
3. Enable the "Use Domino Attachment and Object Service" option.
   
   NOTE: If the database is a mail file or MAIL.BOX file, make sure that the "Use LZ1 compression for attachments" option (the default) is enabled.

To mark multiple selected databases for attachment consolidation, follow these steps:

1. From the Lotus Domino Administrator, choose Files - Advanced Properties.
2. Enable the "Use Domino Attachment and Object Service"option. See figure 3.
   
   NOTE: If the databases are all mail files or MAIL.BOX files, make sure the "Use LZ1 compression for attachments" option (the default) is enabled.

Figure 3. Advanced options

Back to top

Managing the Lotus Domino attachment object store
As the DAOS repository grows to accommodate a greater number of attachments, multiple subdirectories are automatically created on the DAOS base path to balance the data load.
NOTE: There is never any need to modify, move, or delete subdirectories within the repository directory (by default, notesdatadirectory/DAOS) or to modify the NLO files within them.
If the amount of additional space on the current disk is limited, you can move the repository directory to a separate disk that has more space. Also, if the directory is on a separate drive and becomes corrupted due to a server problem, you can specify a new DAOS base path in a new location (for example, on a different drive) and restore the contents of the directory from a backup.
To move or restore the DAOS repository, follow these steps:

1. Do one of the following to move or restore the DAOS repository:
   
   • To move the DAOS repository directory, create the new directory in the new location, and use your operating system to copy the entire contents of the existing notesdatadirectory/DAOS directory to it.
   • To restore the DAOS repository directory from your backup file, copy the backed-up contents of the notesdatadirectory/DAOS directory to the new file location.
2. From the Lotus Domino Administrator, click the Configuration tab and then expand the Server section.
3. Select the server document to be edited, and then click Edit Server.
4. Click the DAOS tab.
5. In the DAOS base path field, specify the new location for the directory. Save the Server document.
6. Restart the Lotus Domino server.

Back to top

Deleting DAOS unreferenced objects
Deferred deletion of DAOS unreferenced objects occurs on a server-wide basis by enabling the "Defer object deletion for n days" setting on the DAOS tab in the server document.
A DAOS object is considered unreferenced only when the last reference to it in any document that refers to it is deleted. With deferred deletion enabled, the object is retained for an additional grace period determined by the server document setting. This extra time aids in recovery if restoring the object becomes necessary.
You can run DAOS Prune with a "number of days old" argument to override the setting in the server document.
Syntax: DAOS Prune [number of days old]
This argument deletes all DAOS unreferenced objects that are older than the "number of days old" argument specified.
The command is useful for cleaning up all objects if you want to do that independently from the default deferred deletion interval.
NOTE: If you run DAOS Prune and enter 0 for number of days old, it removes all unreferenced objects immediately.

Back to top

Backing up servers that participate in DAOS
When databases on a server are set to participate in attachment consolidation, backing up the server requires an additional step of backing up all NLO files stored in the DAOS repository. Such files are not usually subject to frequent modification, so after an initial backup, you can perform incremental backups without much trouble or time.
NOTE: This procedure assumes that, in case of failure, you plan to restore the files to the originating Lotus Domino server. If NLO files have been encrypted with the originating server's key, they are readable only when restored to the originating Lotus Domino server.

1. If you are using deferred deletion with DAOS, set the interval to longer than the interval between your backups. For example, if you back up weekly, specify eight days for the setting "Defer deletion of DAOS objects n days" in the server document.
2. Back up NSF files on the server using a backup utility that is compatible with NSF files. The utility must be able to use the backup and recovery methods of the Lotus Domino C API Toolkit.
3. Back up the DAOSCAT.NSF and DAOS.CFG files. These files are located in the data directory.
4. Back up all NLO files in your DAOS repository. You can use any flat file backup utility of your choice (such as Tivoli® Storage Manager). If DAOS has created subdirectories, maintain the directory hierarchy in your backup.
5. After the first backup of the DAOS repository, perform incremental backups as desired of both NSF and NLO files.
6. (Optional) It is highly recommended that you archive any transaction logs so that changes that occurred after the last backup can be replayed for the most complete restoration of data.

Back to top

Summary of key points
These are the key points that you should remember about DAOS:

• DAOS helps you adopt green computing practices by using less disk space, CPU utilization, and I/O.
• Access to DAOS objects is transparent to the Lotus Domino server and applications:
  
  • Lotus Notes large objects are encrypted and controlled by Lotus Notes ACLs.
  • DAOS is transparent to C API calls, agents, and LotusScript®.
• DAOS improves mail performance:
  
  • Attachments are written once for each server.
  • Additional users get only a copy of the reference.
• Significantly fewer bytes are pushed from a DAOS-enabled MAIL.BOX:
  
  • The same attachment can go to multiple recipients on the same server.
  • MAIL.BOX does not need to be expanded to hold ephemeral attachments.
  • When MAIL.BOX is done routing the mail, it deletes its copy of the attachment, effectively transferring the reference to the mail recipients.
• DAOS improves compact performance by avoiding the need to move large objects during the compaction process where only a small reference is moved.
• DAOS requires transaction logging.
• DAOS objects count against quota and are reported as part of the file size.
• DAOS enables a large reduction in incremental backup costs because the DAOS repository isolates large blocks of data into separate, unchanging files.
• DAOS is compatible with existing databases and servers. For non-DAOS-enabled databases and pass-through requests, the entire attachment is copied.