Chinh Do

Include File Operations in Your Transactions Today with IEnlistmentNotification

25th August 2008

Include File Operations in Your Transactions Today with IEnlistmentNotification

Would it be nice if we can do something like this in our applications?

// Wrap a file copy and a database insert in the same transaction
TxFileManager fileMgr = new TxFileManager();
using (TransactionScope scope1 = new TransactionScope())
{
    // Copy a file
    fileMgr.CopyFile(srcFileName, destFileName);

    // Insert a database record
    dbMgr.ExecuteNonQuery(insertSql);

    scope1.Complete();
}

With the rich support currently available for transactional programming, one may find it rather surprising that the most basic type of program operation, file manipulation (copy file, move file, delete file, write to file, etc.), are typically not transactional in today’s applications.

I am sure the main reason for this situation is lack of support for transactions in the underlying file systems. While Microsoft is bringing us Transactional NTFS (TxF) in Vista and Windows Server 2008, most corporate IT applications are still deployed to Windows 2003 or earlier. While I can’t wait to be able to use TxF, I have applications that have to be completed today!

While searching for a solution, I came across several articles describing the use of IEnlistmentNotification to implement your own resource manager and participate in a System.Transactions.Transaction. However, a complete working code example was nowhere to be found. Well, I guess it’s my turn to contribute. I hereby present to you: Chinh Do’s Transactional File Manager.

Here are my basic requirements for a Transactional File Manager:

  • Works with .NET 2.0′s System.Transactions.
  • Ability to wrap the following file operations in a transaction:
    • Creating a file.
    • Deleting a file.
    • Copying a file.
    • Moving a file.
    • Writing data to a file.
    • Appending data to a file.
    • Creating a directory.
  • Ability to take a snapshot of a file (and restore it to the snapshot state later if required). The snapshot feature allows the inclusion of 3rd-party file operations in your transaction.
  • Thread-safe.

IEnlistmentNotification and ThreadStatic Attribute

Implementing IEnlistmentNotification is harder that it looks… at least for me it was. It’s not enough to just store a list of file operations. Because transactions can be nested and started from different threads; when rolling back, we have to make sure to only include the correct operations for the current Transaction. At first glance, it looks like we should be able to use the LocalIdentifier property (Transaction.TransactionInformation.LocalIdentifier) to identify the current transaction. However, further investigation reveals that Transaction.Current is not available in our various IEnlistmentNotification methods.

As it turned out, the little known but very cool ThreadStatic attribute fits the bill very well. Since the scope of a TransactionScope spans all operations on the same thread inside the TransactionScope block (excluding nested, new Transactions), ThreadStatic gives us an easy way to track that data.

/// <summary>Dictionary of transaction participants for the current thread.</summary>
[ThreadStatic] private static Dictionary<string, TxParticipant> _participants;

In the initial version of my Transactional File Manager class (TxFileManager), I made the mistake of trying to implement IEnlistmentNotification in the main TxFileManager class. I had all kinds of difficulty trying to sort out different transactions/threads. Once I started to split to IEnlistmentNotification implementation into its own nested class (TxParticipant), everything became much cleaner. In the main class, all I have to do is to maintain a Dictionary<T, T> of TxEnlistment objects, which implement IEnlistmentNotification. Each TxEnlistment object would be responsible for handling a separate Transaction. Once that is in place, everything else was like pretty much a walk through the park.

IEnlistmentNotification.Commit

Since my Resource Manager always performs operations immediately, there is really nothing to commit, except to clean up temporary files:

public void Commit(Enlistment enlistment)
{
    for (int i = 0; i < _journal.Count; i++)
    {
        _journal[i].CleanUp();
    } 

    _enlisted = false;
    _journal.Clear();
}

IEnlistmentNotification.Rollback

Rolling back is a little bit more complicated. To ensure consistency, we must roll back operations in reverse order.

Another gotcha I ran into is that Rollback is often (if not all the time) called from a different thread from the Transaction thread. Any unhandled exception that occurs in Rollback will cause an AppDomain.CurrentDomain.UnhandledException. To “handle” an UnhandledException, you can either set IgnoreExceptionsInRollback = True or implement an UnhandledExceptionEventHandler.

public void Rollback(Enlistment enlistment)
{
    try
    {
        // Roll back journal items in reverse order
        for (int i = _journal.Count - 1; i >= 0; i--)
        {
            _journal[i].Rollback();
            _journal[i].CleanUp();
        } 

        _enlisted = false;
        _journal.Clear();
    }
    catch (Exception e)
    {
        if (IgnoreExceptionsInRollback)
        {
            EventLog.WriteEntry(GetType().FullName, "Failed to rollback."
                + Environment.NewLine + e.ToString(), EventLogEntryType.Warning);
        }
        else
        {
            throw new TransactionException("Failed to roll back.", e);
        }
    }
    finally
    {
        _enlisted = false;
        if (_journal != null)
        {
            _journal.Clear();
        }
    } 

    enlistment.Done();
}

Test Driven Development /Unit Testing

What does TDD have to do with this? It just happens that if you do Test Driven Development, Transactional File Manager can make testing classes that perform file operations much more convenient. In conjunction with a mocking framework such as Rhino Mocks, you can easily test the class functionality without having to read/write to actual files.

MockRepository mocks = new MockRepository();
MyClass1 target = new MyClass1();
Target.FileManager = new TxFileManager();
using (mocks.Record())
{
    Expect.Call(target.FileManager.ReadAllText()).Return("abc");
}
using (mocks.Playback())
{
    target.DoWork();
}

Shortcomings

Here are the known shortcomings of my Transactional File Manager:

  • Oher processes and transactions can see pending changes. This effectively makes the Transaction Isolation Level “Read Uncommitted”. This is actually advantageous because it allows external code to participate in our transactions. Without the ability for external code to see “dirty data”, our Transaction File Manager would only be useful in the most narrow of scenarios.
  • There is a performance penalty due to the need to make backups of files involved in the transaction (this is common to all transaction managers). If your process involves working with very large files then using Transactional File Manager may not be practical. In general, transactions should be kept to small and manageable units of work anyway.
  • Only volatile enlistment supported. If the app crashes or is killed, your transaction will be stuck half-way (perhaps durable enlistment will be added in a future version.)

Example 1

// Complete unrealistic example showing how various file operations, including operations done
// by library/3rd party code, can participate in transactions.
IFileManager fileManager = new TxFileManager();
using (TransactionScope scope1 = new TransactionScope())
{
    fileManager.WriteAllText(inFileName, xml);

    // Snapshot allows any file operation to be part of our transaction.
    // All we need to know is the file name.
    XslCompiledTransform xsl = new XslCompiledTransform(true);
    xsl.Load(uri);

    //The statement below tells the TxFileManager to remember the state of this file.
    // So even though XslCompiledTransform has no knowledge of our TxFileManager, the file it creates (outFileName)
    // will still be restored to this state in the event of a rollback.
    fileManager.Snapshot(outFileName);
    xsl.Transform(inFileName, outFileName);

    // write to database 1
    myDb1.ExecuteNonQuery(sql1);

    // write to database 2. The transaction is promoted to a distributed transaction here.
    myDb2.ExecuteNonQuery(sql2);

    // let's delete some files
    for (string fileName in filesToDelete)
    {
        fileManager.Delete(fileName);
    }

    // Just for kicks, let's start a new transaction.
    // Note that we can still use the same fileManager instance. It knows how to sort things out correctly.
    using (TransactionScope scope2 = new TransactionScope(TransactionScopeOptions.RequiresNew))
    {
        fileManager.MoveFile(anotherFile, anotherFileDest);
    }

    // move some files
    for (string fileName in filesToMove)
    {
        fileManager.Move(fileName, GetNewFileName(fileName));
    }

    // Finally, let's create a few temporary files...
    // disk space has to be used for something.
    // The nice thing about FileManager.GetTempFileName is that
    // The temp file will be cleaned up automatically for you when the TransactionScope completes.
    // No more worries about temp files that get left behind.
    for (int i=0; i<10; i++)
    {
        fileManager.WriteAllText(fileManager.GetTempFileName(), "testing 1 2");
    }

    scope1.Complete();
    // In the event an exception occurs, everything done here will be rolled back including the output xsl file.

}

Additional Reading

Updates

  • 12/24/2008 – Version 1.0.1: Fix for memory leak. I fixed the download link above to take you to the new version.
  • 6/8/2010 – Project published to CodePlex. You can download the latest releases from there.

kick it on DotNetKicks.com

This entry was posted on Monday, August 25th, 2008 at 10:32 pm and is filed under Dotnet/.NET - C#, Programming. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

There are currently 33 responses to “Include File Operations in Your Transactions Today with IEnlistmentNotification”

  1. 1 On August 26th, 2008, antlinks said:

    Hey, it appears that the download is coughing up a 404.

  2. 2 On August 26th, 2008, Chinh Do said:

    antlinks:

    Thanks for the note. I’ve fixed the broken link.

  3. 3 On August 29th, 2008, codaholic said:

    Very nice! Thanks for sharing. It would be nice to add the ability to take a snapshot of a whole directory as well.

  4. 4 On September 2nd, 2008, Chinh Do said:

    Codaholic: Thanks for the note. Good idea re directory snapshot.

  5. 5 On September 4th, 2008, Frederic Gos said:

    Hi ;)

    The link to the source code is still broken.

    brgds
    Frederic

  6. 6 On September 4th, 2008, Chinh Do said:

    Frederic: Please try again. I was using a relative link and I guess it worked only sometimes, depending on whether you were on the main page or were looking at the blog post itself. It should work all the time now. Thanks!

  7. 7 On December 24th, 2008, Memory Leak Fix for Transactional File Manager » Chinh Do said:

    [...] found a memory leak in my Transactional File Manager, so here’s version 1.0.1 with a fix for the leak. Click here to download. This entry was [...]

  8. 8 On February 5th, 2009, Wrap Your Unit Tests in Transactions » Chinh Do said:

    [...] my Transactional File Manager, you can also restore the state of the file system in addition to the database. Transactional File [...]

  9. 9 On September 2nd, 2009, Aldara said:

    Aldara…

    Would it be nice if we can do something like this in our applications?// Wrap a file copy and a data [...]…

  10. 10 On February 3rd, 2010, Marouane said:

    would it be possible to include a method for setting ACLs on the files in the transaction ?

  11. 11 On February 3rd, 2010, Chinh Do said:

    Marouane:

    Restoring ACLs (Access Control Lists) when rolling back transactions is not currently supported. In other words, if you make changes to the ACLs as part of your transaction, and then roll it back, the rolled back files may not retain the original ALCs.

    I’ll see if I can add support for it. Seems like a nice feature to have. Do you happen to have code to retrieve and set ALCs?

    Chinh

  12. 12 On March 18th, 2010, Mark said:

    Hi,

    Thanks for a really useful library. I’ve made a few changes to it

    1. If the file operation itself fails then the operation has already been added to the journal so a roll-back is attempted. I’ve change it to add the operation to the journal after it succeeds

    eg

    public void Copy(string sourceFileName, string destFileName, bool overwrite)
    {
    var rollbackFile = new RollbackFile(destFileName);

    File.Copy(sourceFileName, destFileName, overwrite);

    if (_tx != null)
    {
    _journal.Add(rollbackFile);
    Enlist();
    }
    }

    2. Added support to delete directories

    public void Delete(string path)
    {
    if (File.Exists(path))
    {
    var rollbackFile = new RollbackFile(path);

    File.Delete(path);

    if (_tx != null)
    {
    _journal.Add(rollbackFile);
    Enlist();
    }
    }
    else if(Directory.Exists(path))
    {
    var rollbackDirectory = new RollbackDirectory(path);

    Directory.Delete(path);

    if(_tx!=null)
    {
    _journal.Add(rollbackDirectory);
    Enlist();
    }
    }
    }

    3. Changed RollbackDirectory so deleted directories are recreated

    private class RollbackDirectory : RollbackOperation
    {
    public RollbackDirectory(string path)
    {
    _path = path;
    _existed = Directory.Exists(path);
    }

    public override void Rollback()
    {
    if (_existed)
    {
    Directory.CreateDirectory(_path);
    }
    else
    {
    if (Directory.GetFiles(_path).Length == 0 && Directory.GetDirectories(_path).Length == 0)
    {
    // Delete the dir only if it’s empty
    Directory.Delete(_path);
    }
    else
    {
    EventLog.WriteEntry(GetType().FullName, “Failed to delete directory ” + _path + “. Directory was not empty.”, EventLogEntryType.Warning);
    }
    }
    }

    Mark

  13. 13 On March 18th, 2010, Chinh Do said:

    Hi Mark: Thanks for your comment and code. I will incorporate it into my library soon. Chinh

  14. 14 On May 22nd, 2010, espinete said:

    Hi all,

    where is the all complete code ?? Works in Windows XP sp2 and win 2003 r2 ??

    thanks

  15. 15 On May 22nd, 2010, espinete said:

    All code is update ? the project will be in codeplex ?? any other projects using IEnlistementNOtification ?? thanks

  16. 16 On May 22nd, 2010, Chinh Do said:

    Espinete: The latest code is in the download link in the article. You will want to download it, then incorporate the suggested changes by Mark. CodePlex is a good idea… I will look into that. Chinh

  17. 17 On June 7th, 2010, espinete said:

    Thats cool, codeplex power !!! Thanks

  18. 18 On June 8th, 2010, Chinh Do said:

    I have published the code to Codeplex (http://transactionalfilemgr.codeplex.com/). Cheers.

  19. 19 On June 8th, 2010, Transactional File Manager Is Now On CodePlex » Chinh Do said:

    [...] I’ve gone open source with my Transactional File Manager. It’s my first open source collaboration! Check out the CodePlex link here. [...]

  20. 20 On October 27th, 2010, Pratik Kothari said:

    Should I implement IDisposable interface for TxEnlistment class or is it not needed?

  21. 21 On October 24th, 2011, Shukhrat Nekbaev said:

    Hi,

    how about working with network pathes?
    For example following: Directory.Exists(pathTargetProjectFolder) vs fileMgr.DirectoryExists(pathTargetProjectFolder) former returns true, latter false or fileMgr.CreateDirectory(pathTargetProjectFolder) fails with System.ArgumentException: Path cannot be the empty string or all whitespace, but Directory.CreateDirectory(pathTargetProjectFolder) succeedes.

    Application is impersonated.

    Thank you!

  22. 22 On October 24th, 2011, Chinh Do said:

    Shukhrat: Which version of the code are you using? I tested with the latest code (you can download from CodePlex) and it works fine:
    Assert.IsFalse(_target.DirectoryExists(@”\\server\share”));
    Assert.IsTrue(_target.DirectoryExists(@”\\vpc01\test”));

  23. 23 On November 30th, 2011, Ashwin J said:

    While going through this article i also came across ISinglePhaseNotification Interface and also a link to “Optimization using Single Phase Commit and Promotable Single Phase Notification” http://msdn.microsoft.com/en-us/library/ms229980(v=vs.90).aspx
    ,
    Can you let me know when should one go for a Implmenting
    ISinglePhaseNotification Interface vs. IEnlistmentNotification Interface.

    I am currently having a database delete and file delete operations to be performed within a single transaction.

    Thanks for the reply in advance

  24. 24 On November 30th, 2011, Chinh Do said:

    Ashwin: Sorry I don’t have any additional info on ISinglePhaseNotification. But if what you need to do is have database and file ops wrapped in the same transactions, you should be able to use the current release of .NET Transactional File Manager. Is there any issue that prevents you from using that?

  25. 25 On November 30th, 2011, Ashwin J said:

    Just wanted to confirm my understanding that when i use 2 resources 1. Database Connection 2. File I/O inside a single transaction, i need to use IEnlishmentNotification only and i can’t achieve the said fucntionlaity using ISinglePhaseNotification??? so this point is what is currently stopping me from making a decision to whether go with IEnlishmentNotification or ISinglePhaseNotification

  26. 26 On November 30th, 2011, Chinh Do said:

    Ashwin:

    Yes for what you need, IEnlistmentNotification will work.

    ISinglePhaseNotification is to implement single-phase commits. Read more about single-phase and mult-phase commits here: http://msdn.microsoft.com/en-us/library/ckawh9ct(v=vs.80).aspx

    In general, single-phase is more efficient. Two-phase is safer.

  27. 27 On January 7th, 2012, Nick L. said:

    Chinh-
    Great write up and thanks for sharing. I have a need for something similar and your insight is helpful. I hope all is well and happy new year!
    Nick

  28. 28 On April 20th, 2012, Jan said:

    Hello, nice article, i found it a little bit to late :(
    how have you implemented the inDoubt method?

  29. 29 On April 20th, 2012, Chinh Do said:

    Jan: Thanks. No we did not implement InDoubt. If you do implement it please consider contributing the code to the CodePlex project. Thanksa

  30. 30 On June 22nd, 2012, Sunny said:

    Does it stubborn enough to withstand file with out getting currupted when a sudden power failure happens when writing data to the file.

    In my case, I am writing some data using FileStream to a file, and while writing wantedly will remove power cable suddenly. After restart the file contains some nul chars embedded instead of the actual data. Can this be avoided with this or else any other way to handle this scenario. Pls suggest/update me

    Thanks in advance.

    Sunny

  31. 31 On June 22nd, 2012, Chinh Do said:

    Sunny:

    That’s currently not supported by this framework. In theory, it can be done by implementing a separate transaction manager which is aware of the state of various transactions and can roll back incomplete transactions like the one in your example.

    Chinh

  32. 32 On February 21st, 2014, Me said:

    I don’t see any source code in here:
    https://transactionalfilemgr.codeplex.com/SourceControl/list/changesets
    :(

  33. 33 On March 27th, 2014, Chinh Do said:

    Hi: Sorry for the slow response… gmail tabs hid my notifications. To get source code, you can either do a Clone or you can browse the files here https://transactionalfilemgr.codeplex.com/SourceControl/latest.

    Chinh

Leave a Comment

*