Include File Operations in Your Transactions Today with IEnlistmentNotification

Would it be nice if we can do something like this in our applications?

// Wrap a file copy and a database insert in the same transaction
TxFileManager fileMgr = new TxFileManager();
using (TransactionScope scope1 = new TransactionScope())
{
    // Copy a file
    fileMgr.CopyFile(srcFileName, destFileName);

    // Insert a database record
    dbMgr.ExecuteNonQuery(insertSql);

    scope1.Complete();
}

With the rich support currently available for transactional programming, one may find it rather surprising that the most basic type of program operation, file manipulation (copy file, move file, delete file, write to file, etc.), are typically not transactional in today’s applications.

I am sure the main reason for this situation is lack of support for transactions in the underlying file systems. While Microsoft is bringing us Transactional NTFS (TxF) in Vista and Windows Server 2008, most corporate IT applications are still deployed to Windows 2003 or earlier. While I can’t wait to be able to use TxF, I have applications that have to be completed today!

While searching for a solution, I came across several articles describing the use of IEnlistmentNotification to implement your own resource manager and participate in a System.Transactions.Transaction. However, a complete working code example was nowhere to be found. Well, I guess it’s my turn to contribute. I hereby present to you: Chinh Do’s Transactional File Manager.

Here are my basic requirements for a Transactional File Manager:

  • Works with .NET 2.0’s System.Transactions.
  • Ability to wrap the following file operations in a transaction:
    • Creating a file.
    • Deleting a file.
    • Copying a file.
    • Moving a file.
    • Writing data to a file.
    • Appending data to a file.
    • Creating a directory.
  • Ability to take a snapshot of a file (and restore it to the snapshot state later if required). The snapshot feature allows the inclusion of 3rd-party file operations in your transaction.
  • Thread-safe.

IEnlistmentNotification and ThreadStatic Attribute

Implementing IEnlistmentNotification is harder that it looks… at least for me it was. It’s not enough to just store a list of file operations. Because transactions can be nested and started from different threads; when rolling back, we have to make sure to only include the correct operations for the current Transaction. At first glance, it looks like we should be able to use the LocalIdentifier property (Transaction.TransactionInformation.LocalIdentifier) to identify the current transaction. However, further investigation reveals that Transaction.Current is not available in our various IEnlistmentNotification methods.

As it turned out, the little known but very cool ThreadStatic attribute fits the bill very well. Since the scope of a TransactionScope spans all operations on the same thread inside the TransactionScope block (excluding nested, new Transactions), ThreadStatic gives us an easy way to track that data.

/// <summary>Dictionary of transaction participants for the current thread.</summary>
[ThreadStatic] private static Dictionary<string, TxParticipant> _participants;

In the initial version of my Transactional File Manager class (TxFileManager), I made the mistake of trying to implement IEnlistmentNotification in the main TxFileManager class. I had all kinds of difficulty trying to sort out different transactions/threads. Once I started to split to IEnlistmentNotification implementation into its own nested class (TxParticipant), everything became much cleaner. In the main class, all I have to do is to maintain a Dictionary<T, T> of TxEnlistment objects, which implement IEnlistmentNotification. Each TxEnlistment object would be responsible for handling a separate Transaction. Once that is in place, everything else was like pretty much a walk through the park.

IEnlistmentNotification.Commit

Since my Resource Manager always performs operations immediately, there is really nothing to commit, except to clean up temporary files:

public void Commit(Enlistment enlistment)
{
    for (int i = 0; i < _journal.Count; i++)
    {
        _journal[i].CleanUp();
    } 

    _enlisted = false;
    _journal.Clear();
}

IEnlistmentNotification.Rollback

Rolling back is a little bit more complicated. To ensure consistency, we must roll back operations in reverse order.

Another gotcha I ran into is that Rollback is often (if not all the time) called from a different thread from the Transaction thread. Any unhandled exception that occurs in Rollback will cause an AppDomain.CurrentDomain.UnhandledException. To “handle” an UnhandledException, you can either set IgnoreExceptionsInRollback = True or implement an UnhandledExceptionEventHandler.

public void Rollback(Enlistment enlistment)
{
    try
    {
        // Roll back journal items in reverse order
        for (int i = _journal.Count - 1; i >= 0; i--)
        {
            _journal[i].Rollback();
            _journal[i].CleanUp();
        } 

        _enlisted = false;
        _journal.Clear();
    }
    catch (Exception e)
    {
        if (IgnoreExceptionsInRollback)
        {
            EventLog.WriteEntry(GetType().FullName, "Failed to rollback."
                + Environment.NewLine + e.ToString(), EventLogEntryType.Warning);
        }
        else
        {
            throw new TransactionException("Failed to roll back.", e);
        }
    }
    finally
    {
        _enlisted = false;
        if (_journal != null)
        {
            _journal.Clear();
        }
    } 

    enlistment.Done();
}

Test Driven Development /Unit Testing

What does TDD have to do with this? It just happens that if you do Test Driven Development, Transactional File Manager can make testing classes that perform file operations much more convenient. In conjunction with a mocking framework such as Rhino Mocks, you can easily test the class functionality without having to read/write to actual files.

MockRepository mocks = new MockRepository();
MyClass1 target = new MyClass1();
Target.FileManager = new TxFileManager();
using (mocks.Record())
{
    Expect.Call(target.FileManager.ReadAllText()).Return("abc");
}
using (mocks.Playback())
{
    target.DoWork();
}

Shortcomings

Here are the known shortcomings of my Transactional File Manager:

  • Oher processes and transactions can see pending changes. This effectively makes the Transaction Isolation Level “Read Uncommitted”. This is actually advantageous because it allows external code to participate in our transactions. Without the ability for external code to see “dirty data”, our Transaction File Manager would only be useful in the most narrow of scenarios.
  • There is a performance penalty due to the need to make backups of files involved in the transaction (this is common to all transaction managers). If your process involves working with very large files then using Transactional File Manager may not be practical. In general, transactions should be kept to small and manageable units of work anyway.
  • Only volatile enlistment supported. If the app crashes or is killed, your transaction will be stuck half-way (perhaps durable enlistment will be added in a future version.)

Example 1

// Complete unrealistic example showing how various file operations, including operations done
// by library/3rd party code, can participate in transactions.
IFileManager fileManager = new TxFileManager();
using (TransactionScope scope1 = new TransactionScope())
{
    fileManager.WriteAllText(inFileName, xml);

    // Snapshot allows any file operation to be part of our transaction.
    // All we need to know is the file name.
    XslCompiledTransform xsl = new XslCompiledTransform(true);
    xsl.Load(uri);

    //The statement below tells the TxFileManager to remember the state of this file.
    // So even though XslCompiledTransform has no knowledge of our TxFileManager, the file it creates (outFileName)
    // will still be restored to this state in the event of a rollback.
    fileManager.Snapshot(outFileName);
    xsl.Transform(inFileName, outFileName);

    // write to database 1
    myDb1.ExecuteNonQuery(sql1);

    // write to database 2. The transaction is promoted to a distributed transaction here.
    myDb2.ExecuteNonQuery(sql2);

    // let's delete some files
    for (string fileName in filesToDelete)
    {
        fileManager.Delete(fileName);
    }

    // Just for kicks, let's start a new transaction.
    // Note that we can still use the same fileManager instance. It knows how to sort things out correctly.
    using (TransactionScope scope2 = new TransactionScope(TransactionScopeOptions.RequiresNew))
    {
        fileManager.MoveFile(anotherFile, anotherFileDest);
    }

    // move some files
    for (string fileName in filesToMove)
    {
        fileManager.Move(fileName, GetNewFileName(fileName));
    }

    // Finally, let's create a few temporary files...
    // disk space has to be used for something.
    // The nice thing about FileManager.GetTempFileName is that
    // The temp file will be cleaned up automatically for you when the TransactionScope completes.
    // No more worries about temp files that get left behind.
    for (int i=0; i<10; i++)
    {
        fileManager.WriteAllText(fileManager.GetTempFileName(), "testing 1 2");
    }

    scope1.Complete();
    // In the event an exception occurs, everything done here will be rolled back including the output xsl file.

}

Additional Reading

Updates

  • 12/24/2008 – Version 1.0.1: Fix for memory leak. I fixed the download link above to take you to the new version.
  • 6/8/2010 – Project published to CodePlex. You can download the latest releases from there.

kick it on DotNetKicks.com

37 Replies to “Include File Operations in Your Transactions Today with IEnlistmentNotification”

  1. Frederic: Please try again. I was using a relative link and I guess it worked only sometimes, depending on whether you were on the main page or were looking at the blog post itself. It should work all the time now. Thanks!

  2. Pingback: Aldara
  3. Marouane:

    Restoring ACLs (Access Control Lists) when rolling back transactions is not currently supported. In other words, if you make changes to the ACLs as part of your transaction, and then roll it back, the rolled back files may not retain the original ALCs.

    I’ll see if I can add support for it. Seems like a nice feature to have. Do you happen to have code to retrieve and set ALCs?

    Chinh

  4. Hi,

    Thanks for a really useful library. I’ve made a few changes to it

    1. If the file operation itself fails then the operation has already been added to the journal so a roll-back is attempted. I’ve change it to add the operation to the journal after it succeeds

    eg

    public void Copy(string sourceFileName, string destFileName, bool overwrite)
    {
    var rollbackFile = new RollbackFile(destFileName);

    File.Copy(sourceFileName, destFileName, overwrite);

    if (_tx != null)
    {
    _journal.Add(rollbackFile);
    Enlist();
    }
    }

    2. Added support to delete directories

    public void Delete(string path)
    {
    if (File.Exists(path))
    {
    var rollbackFile = new RollbackFile(path);

    File.Delete(path);

    if (_tx != null)
    {
    _journal.Add(rollbackFile);
    Enlist();
    }
    }
    else if(Directory.Exists(path))
    {
    var rollbackDirectory = new RollbackDirectory(path);

    Directory.Delete(path);

    if(_tx!=null)
    {
    _journal.Add(rollbackDirectory);
    Enlist();
    }
    }
    }

    3. Changed RollbackDirectory so deleted directories are recreated

    private class RollbackDirectory : RollbackOperation
    {
    public RollbackDirectory(string path)
    {
    _path = path;
    _existed = Directory.Exists(path);
    }

    public override void Rollback()
    {
    if (_existed)
    {
    Directory.CreateDirectory(_path);
    }
    else
    {
    if (Directory.GetFiles(_path).Length == 0 && Directory.GetDirectories(_path).Length == 0)
    {
    // Delete the dir only if it’s empty
    Directory.Delete(_path);
    }
    else
    {
    EventLog.WriteEntry(GetType().FullName, “Failed to delete directory ” + _path + “. Directory was not empty.”, EventLogEntryType.Warning);
    }
    }
    }

    Mark

  5. All code is update ? the project will be in codeplex ?? any other projects using IEnlistementNOtification ?? thanks

  6. Espinete: The latest code is in the download link in the article. You will want to download it, then incorporate the suggested changes by Mark. CodePlex is a good idea… I will look into that. Chinh

  7. Hi,

    how about working with network pathes?
    For example following: Directory.Exists(pathTargetProjectFolder) vs fileMgr.DirectoryExists(pathTargetProjectFolder) former returns true, latter false or fileMgr.CreateDirectory(pathTargetProjectFolder) fails with System.ArgumentException: Path cannot be the empty string or all whitespace, but Directory.CreateDirectory(pathTargetProjectFolder) succeedes.

    Application is impersonated.

    Thank you!

  8. Shukhrat: Which version of the code are you using? I tested with the latest code (you can download from CodePlex) and it works fine:
    Assert.IsFalse(_target.DirectoryExists(@”\\server\share”));
    Assert.IsTrue(_target.DirectoryExists(@”\\vpc01\test”));

  9. While going through this article i also came across ISinglePhaseNotification Interface and also a link to “Optimization using Single Phase Commit and Promotable Single Phase Notification” http://msdn.microsoft.com/en-us/library/ms229980(v=vs.90).aspx
    ,
    Can you let me know when should one go for a Implmenting
    ISinglePhaseNotification Interface vs. IEnlistmentNotification Interface.

    I am currently having a database delete and file delete operations to be performed within a single transaction.

    Thanks for the reply in advance

  10. Ashwin: Sorry I don’t have any additional info on ISinglePhaseNotification. But if what you need to do is have database and file ops wrapped in the same transactions, you should be able to use the current release of .NET Transactional File Manager. Is there any issue that prevents you from using that?

  11. Just wanted to confirm my understanding that when i use 2 resources 1. Database Connection 2. File I/O inside a single transaction, i need to use IEnlishmentNotification only and i can’t achieve the said fucntionlaity using ISinglePhaseNotification??? so this point is what is currently stopping me from making a decision to whether go with IEnlishmentNotification or ISinglePhaseNotification

  12. Chinh-
    Great write up and thanks for sharing. I have a need for something similar and your insight is helpful. I hope all is well and happy new year!
    Nick

  13. Does it stubborn enough to withstand file with out getting currupted when a sudden power failure happens when writing data to the file.

    In my case, I am writing some data using FileStream to a file, and while writing wantedly will remove power cable suddenly. After restart the file contains some nul chars embedded instead of the actual data. Can this be avoided with this or else any other way to handle this scenario. Pls suggest/update me

    Thanks in advance.

    Sunny

  14. Sunny:

    That’s currently not supported by this framework. In theory, it can be done by implementing a separate transaction manager which is aware of the state of various transactions and can roll back incomplete transactions like the one in your example.

    Chinh

Leave a Reply

Your email address will not be published. Required fields are marked *