skip to Main Content

Include File Operations in Your Transactions Today with IEnlistmentNotification

Hi 2021 visitors 🙂 The current version of this library is available as a Nuget package and is also on GitHub. If you have any issues with this code please open an issue on GitHub.

Would it be nice if we can do something like this in our applications?

// Wrap a file copy and a database insert in the same transaction
TxFileManager fileMgr = new TxFileManager();
using (TransactionScope scope1 = new TransactionScope())
{
    // Copy a file
    fileMgr.CopyFile(srcFileName, destFileName);

    // Insert a database record
    dbMgr.ExecuteNonQuery(insertSql);

    scope1.Complete();
}

With the rich support currently available for transactional programming, one may find it rather surprising that the most basic type of program operation, file manipulation (copy file, move file, delete file, write to file, etc.), are typically not transactional in today’s applications.

I am sure the main reason for this situation is lack of support for transactions in the underlying file systems. While Microsoft is bringing us Transactional NTFS (TxF) in Vista and Windows Server 2008, most corporate IT applications are still deployed to Windows 2003 or earlier. While I can’t wait to be able to use TxF, I have applications that have to be completed today!

While searching for a solution, I came across several articles describing the use of IEnlistmentNotification to implement your own resource manager and participate in a System.Transactions.Transaction. However, a complete working code example was nowhere to be found. Well, I guess it’s my turn to contribute. I hereby present to you: Chinh Do’s Transactional File Manager.

Here are my basic requirements for a Transactional File Manager:

  • Works with .NET 2.0’s System.Transactions.
  • Ability to wrap the following file operations in a transaction:
    • Creating a file.
    • Deleting a file.
    • Copying a file.
    • Moving a file.
    • Writing data to a file.
    • Appending data to a file.
    • Creating a directory.
  • Ability to take a snapshot of a file (and restore it to the snapshot state later if required). The snapshot feature allows the inclusion of 3rd-party file operations in your transaction.
  • Thread-safe.

IEnlistmentNotification and ThreadStatic Attribute

Implementing IEnlistmentNotification is harder that it looks… at least for me it was. It’s not enough to just store a list of file operations. Because transactions can be nested and started from different threads; when rolling back, we have to make sure to only include the correct operations for the current Transaction. At first glance, it looks like we should be able to use the LocalIdentifier property (Transaction.TransactionInformation.LocalIdentifier) to identify the current transaction. However, further investigation reveals that Transaction.Current is not available in our various IEnlistmentNotification methods.

As it turned out, the little known but very cool ThreadStatic attribute fits the bill very well. Since the scope of a TransactionScope spans all operations on the same thread inside the TransactionScope block (excluding nested, new Transactions), ThreadStatic gives us an easy way to track that data.

/// <summary>Dictionary of transaction participants for the current thread.</summary>
[ThreadStatic] private static Dictionary<string, TxParticipant> _participants;

In the initial version of my Transactional File Manager class (TxFileManager), I made the mistake of trying to implement IEnlistmentNotification in the main TxFileManager class. I had all kinds of difficulty trying to sort out different transactions/threads. Once I started to split to IEnlistmentNotification implementation into its own nested class (TxParticipant), everything became much cleaner. In the main class, all I have to do is to maintain a Dictionary<T, T> of TxEnlistment objects, which implement IEnlistmentNotification. Each TxEnlistment object would be responsible for handling a separate Transaction. Once that is in place, everything else was like pretty much a walk through the park.

IEnlistmentNotification.Commit

Since my Resource Manager always performs operations immediately, there is really nothing to commit, except to clean up temporary files:

public void Commit(Enlistment enlistment)
{
    for (int i = 0; i < _journal.Count; i++)
    {
        _journal[i].CleanUp();
    } 

    _enlisted = false;
    _journal.Clear();
}

IEnlistmentNotification.Rollback

Rolling back is a little bit more complicated. To ensure consistency, we must roll back operations in reverse order.

Another gotcha I ran into is that Rollback is often (if not all the time) called from a different thread from the Transaction thread. Any unhandled exception that occurs in Rollback will cause an AppDomain.CurrentDomain.UnhandledException. To “handle” an UnhandledException, you can either set IgnoreExceptionsInRollback = True or implement an UnhandledExceptionEventHandler.

public void Rollback(Enlistment enlistment)
{
    try
    {
        // Roll back journal items in reverse order
        for (int i = _journal.Count - 1; i >= 0; i--)
        {
            _journal[i].Rollback();
            _journal[i].CleanUp();
        } 

        _enlisted = false;
        _journal.Clear();
    }
    catch (Exception e)
    {
        if (IgnoreExceptionsInRollback)
        {
            EventLog.WriteEntry(GetType().FullName, "Failed to rollback."
                + Environment.NewLine + e.ToString(), EventLogEntryType.Warning);
        }
        else
        {
            throw new TransactionException("Failed to roll back.", e);
        }
    }
    finally
    {
        _enlisted = false;
        if (_journal != null)
        {
            _journal.Clear();
        }
    } 

    enlistment.Done();
}

Test Driven Development /Unit Testing

What does TDD have to do with this? It just happens that if you do Test Driven Development, Transactional File Manager can make testing classes that perform file operations much more convenient. In conjunction with a mocking framework such as Rhino Mocks, you can easily test the class functionality without having to read/write to actual files.

MockRepository mocks = new MockRepository();
MyClass1 target = new MyClass1();
Target.FileManager = new TxFileManager();
using (mocks.Record())
{
    Expect.Call(target.FileManager.ReadAllText()).Return("abc");
}
using (mocks.Playback())
{
    target.DoWork();
}

Shortcomings

Here are the known shortcomings of my Transactional File Manager:

  • Oher processes and transactions can see pending changes. This effectively makes the Transaction Isolation Level “Read Uncommitted”. This is actually advantageous because it allows external code to participate in our transactions. Without the ability for external code to see “dirty data”, our Transaction File Manager would only be useful in the most narrow of scenarios.
  • There is a performance penalty due to the need to make backups of files involved in the transaction (this is common to all transaction managers). If your process involves working with very large files then using Transactional File Manager may not be practical. In general, transactions should be kept to small and manageable units of work anyway.
  • Only volatile enlistment supported. If the app crashes or is killed, your transaction will be stuck half-way (perhaps durable enlistment will be added in a future version.)

Example 1

// Complete unrealistic example showing how various file operations, including operations done
// by library/3rd party code, can participate in transactions.
IFileManager fileManager = new TxFileManager();
using (TransactionScope scope1 = new TransactionScope())
{
    fileManager.WriteAllText(inFileName, xml);

    // Snapshot allows any file operation to be part of our transaction.
    // All we need to know is the file name.
    XslCompiledTransform xsl = new XslCompiledTransform(true);
    xsl.Load(uri);

    //The statement below tells the TxFileManager to remember the state of this file.
    // So even though XslCompiledTransform has no knowledge of our TxFileManager, the file it creates (outFileName)
    // will still be restored to this state in the event of a rollback.
    fileManager.Snapshot(outFileName);
    xsl.Transform(inFileName, outFileName);

    // write to database 1
    myDb1.ExecuteNonQuery(sql1);

    // write to database 2. The transaction is promoted to a distributed transaction here.
    myDb2.ExecuteNonQuery(sql2);

    // let's delete some files
    for (string fileName in filesToDelete)
    {
        fileManager.Delete(fileName);
    }

    // Just for kicks, let's start a new transaction.
    // Note that we can still use the same fileManager instance. It knows how to sort things out correctly.
    using (TransactionScope scope2 = new TransactionScope(TransactionScopeOptions.RequiresNew))
    {
        fileManager.MoveFile(anotherFile, anotherFileDest);
    }

    // move some files
    for (string fileName in filesToMove)
    {
        fileManager.Move(fileName, GetNewFileName(fileName));
    }

    // Finally, let's create a few temporary files...
    // disk space has to be used for something.
    // The nice thing about FileManager.GetTempFileName is that
    // The temp file will be cleaned up automatically for you when the TransactionScope completes.
    // No more worries about temp files that get left behind.
    for (int i=0; i<10; i++)
    {
        fileManager.WriteAllText(fileManager.GetTempFileName(), "testing 1 2");
    }

    scope1.Complete();
    // In the event an exception occurs, everything done here will be rolled back including the output xsl file.

}

Additional Reading

Updates

  • 12/24/2008 – Version 1.0.1: Fix for memory leak. I fixed the download link above to take you to the new version.
  • 6/8/2010 – Project published to CodePlex. You can download the latest releases from there.
  • 3/18/2020 – Wow almost 10 years later – I migrated the code from CodePlex to GitHub. This library is also on Nuget here.

Windows Mobile 6.1 for Samsung SCH-i760

Good news for i760 owners: Windows Mobile 6.1 update is now available.

This update fixes one major annoyance: support for SDHC cards with more than 2GB. I use my i760 as a music player and it’s kind of tough to have to fit my music selection into 2GB (or 4GB if you use the hack but didn’t want to use a hack).

Some features of note in WM 6.1:

  • Support for SDHC cards  beyond 2GB. 8GB cards seem to be working fine for people.
  • Threaded SMS reader.
  • View YouTube videos (m.youtube.com).

Download the update from Verizon here.

More information here.

Verizon SCH-i760 Upgrade Tool

A word of warning:  the upgrade utility provided by Verizon/Samsung looks like a major piece of rushware. Many people reported no problems with the upgrade, but others reported of bricked phones, partially upgraded phones, and having to try the upgrade process multiple times to get it to work.

Finds of the Week – June 17, 2008

.NET, C#

General Development

Software, Tools, etc.

Gaming

Something Different

Finds of the Week – June 9, 2008

.NET/C#

  • Microsoft project code named “Velocity” is a distributed in-memory caching platform that provides .NET applications with high-speed access, scaling, and high availability to application data. Download the Community Preview here.
  • Danny Simmons enumerated various reasons for using Entity Framework.
  • If your netMsmq WCF service shows signs of a handles leak, you may want to make sure you have .NET Framework 3.0 Service Pack 1 installed.
  • Bryan wrote a timely article on TDD Tips: Test Naming Conventions & Guidelines.
  • Microsoft announced the release of Microsoft Source Analysis for C#.
    “Source Analysis is similar in many ways to Microsoft Code Analysis (specifically FxCop), but there are some important distinctions. FxCop performs its analysis on compiled binaries, while Source Analysis analyzes the source code directly. For this reason, Code Analysis focuses more on the design of the code, while Source Analysis focuses on layout, readability and documentation. Most of that information is stripped away during the compilation process, and thus cannot be analyzed by FxCop.”

General Programming

Tools

  • The Query command line utility displays active Terminal Service/Remote Desktop sessions, among other things. This replaces the qwinsta utility.

Something Different

Finds of the Week – May 31, 2008

Programming

.NET/C#

Software and Tools

  • Google Maps Street View is in Richmond.

    Google Street View Richmond Virginia Goo

    It looks like the Street View images were taken around September 2007 because according to IMDB, the movie Game Plan was playing in the US starting 9/23/2007:

    Google Street View Richmond

  • Google Maps adds user-created photos, videos, maps. Via CNET.

    Google Maps Goog

Something Different

Splitting a Generic List<T> into Multiple Chunks

Greetings visitor from the year 2020! You can get the latest code for this from my Github repo here. Thanks for visiting.

“Chunking” is the technique used to break large amount of work into smaller and manageable parts. Here are a few reasons I can think of why you want to chunk, especially in a batch process where you have to process large number of items:

  • Manage/minimize peak memory requirement.
  • During failures, the entire process can resume at the last failure point, instead of all the way from the beginning.
  • Take advantage of multiple processors/cores (by having multiple threads, each processing a small chunk).

Here’s a helper method to quickly split a List<T> into chunks:

/// <summary>
/// Splits a <see cref="List{T}"/> into multiple chunks.
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="list">The list to be chunked.</param>
/// <param name="chunkSize">The size of each chunk.</param>
/// <returns>A list of chunks.</returns>
public static List<List<T>> SplitIntoChunks<T>(List<T> list, int chunkSize)
{
    if (chunkSize <= 0)
    {
        throw new ArgumentException("chunkSize must be greater than 0.");
    }

    List<List<T>> retVal = new List<List<T>>();
    int index = 0;
    while (index < list.Count)
    {
        int count = list.Count - index > chunkSize ? chunkSize : list.Count - index;
        retVal.Add(list.GetRange(index, count));

        index += chunkSize;
    }

    return retVal;
}
 

If you want to be more efficient at the cost of readability, the second version below moves the items from the big list into the small chunks, so both types of lists will not need to be in memory at once:

 

/// <summary>
/// Break a <see cref="List{T}"/> into multiple chunks. The <paramref name="list="/> is cleared out and the items are moved
/// into the returned chunks.
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="list">The list to be chunked.</param>
/// <param name="chunkSize">The size of each chunk.</param>
/// <returns>A list of chunks.</returns>
public static List<List<T>> BreakIntoChunks<T>(List<T> list, int chunkSize)
{
    if (chunkSize <= 0)
    {
        throw new ArgumentException("chunkSize must be greater than 0.");
    }

    List<List<T>> retVal = new List<List<T>>();

    while (list.Count > 0)
    {
        int count = list.Count > chunkSize ? chunkSize : list.Count;
        retVal.Add(list.GetRange(0, count));
        list.RemoveRange(0, count);
    }

    return retVal;
}

Finds of the Week – April 6, 2008

Programming

.NET/C#

Windows Mobile/Pocket PC

Gaming

Something a Little Different

Finds of the Week – March 30, 2008

Programming

.NET/C#

PowerShell

Back To Top