Splitting a Generic List<T> into Multiple Chunks

“Chunking” is the technique used to break large amount of work into smaller and manageable parts. Here are a few reasons I can think of why you want to chunk, especially in a batch process where you have to process large number of items:

  • Manage/minimize peak memory requirement.
  • During failures, the entire process can resume at the last failure point, instead of all the way from the beginning.
  • Take advantage of multiple processors/cores (by having multiple threads, each processing a small chunk).

Here’s a helper method to quickly split a List<T> into chunks:

/// <summary>
/// Splits a <see cref="List{T}"/> into multiple chunks.
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="list">The list to be chunked.</param>
/// <param name="chunkSize">The size of each chunk.</param>
/// <returns>A list of chunks.</returns>
public static List<List<T>> SplitIntoChunks<T>(List<T> list, int chunkSize)
{
    if (chunkSize <= 0)
    {
        throw new ArgumentException("chunkSize must be greater than 0.");
    }

    List<List<T>> retVal = new List<List<T>>();
    int index = 0;
    while (index < list.Count)
    {
        int count = list.Count - index > chunkSize ? chunkSize : list.Count - index;
        retVal.Add(list.GetRange(index, count));

        index += chunkSize;
    }

    return retVal;
}
 

If you want to be more efficient at the cost of readability, the second version below moves the items from the big list into the small chunks, so both types of lists will not need to be in memory at once:

 

/// <summary>
/// Break a <see cref="List{T}"/> into multiple chunks. The <paramref name="list="/> is cleared out and the items are moved
/// into the returned chunks.
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="list">The list to be chunked.</param>
/// <param name="chunkSize">The size of each chunk.</param>
/// <returns>A list of chunks.</returns>
public static List<List<T>> BreakIntoChunks<T>(List<T> list, int chunkSize)
{
    if (chunkSize <= 0)
    {
        throw new ArgumentException("chunkSize must be greater than 0.");
    }

    List<List<T>> retVal = new List<List<T>>();

    while (list.Count > 0)
    {
        int count = list.Count > chunkSize ? chunkSize : list.Count;
        retVal.Add(list.GetRange(0, count));
        list.RemoveRange(0, count);
    }

    return retVal;
}

12 Replies to “Splitting a Generic List<T> into Multiple Chunks”

  1. (list.Count – index) gives the number of remaining elements including the index element. So it doesn’t have to be greater than chunkSize, rather greater than or equal.

    I think this line:

    int count = list.Count – index > chunkSize ? chunkSize : list.Count – index;

    should be:

    int count = list.Count – index >= chunkSize ? chunkSize : list.Count – index;

    Thanks for the post – helped me with a problem I was having.

  2. Hi Anjo: I checked my code again and it does work as expected (I did have a pretty comprehensive unit test for it). However, you have very sharp eyes and your version also works just fine. When list.Count – index == chunkSize, either the left side or right side of the equation will get you the same thing.

    Thanks for the comment. It was a good brain excercise.

    Chinh

  3. ///
    /// Paged ForEach
    ///
    ///
    ///
    ///
    ///
    public static void ForEachChunk(this List source, int count, Action<List> action) {
    int nTotal = source.Count();
    var nPageCount = (int)Math.Ceiling((double)(source.Count()) / (double)count);
    for (int i = 0; i < nPageCount; i++) {
    var l = source.Skip(i * count).Take(count).ToList();
    action(l);
    }
    }

  4. here are 2 alternatives that does various things, in various speeds and in various manners:
    … both are extension methods…

    public static class ext
    {
    // Define other methods and classes here
    ///
    /// Break a into multiple chunks. The is cleared out and the items are moved
    /// into the returned chunks.
    ///
    ///
    /// The list to be chunked.
    /// The size of each chunk.
    /// Remove elements from input (reduce memory use)
    /// A list of chunks.
    public static IEnumerable<List> BreakIntoChunks(this List list, int chunkSize = 10, bool remove = false)
    {
    if (chunkSize = list.Count)
    yield return list;
    else
    {
    if(remove)
    {
    while (list.Count > 0)
    {
    int count = list.Count > chunkSize ? chunkSize : list.Count;
    var ret = list.GetRange(0, count);
    list.RemoveRange(0, count);
    yield return ret;
    }
    }
    else
    {
    int count = 0, max = list.Count – (list.Count % chunkSize);
    do
    {
    yield return list.GetRange(count, chunkSize);
    count += chunkSize;
    } while (count < max);

    if(count < list.Count)
    yield return list.GetRange(count, (list.Count % chunkSize));
    }
    }
    }

    //Quick and dirty…
    public static List<List> Split(this List source, int size)
    {
    if (size new { Index = i, Value = x })
    .GroupBy(x => x.Index / size)
    .Select(x => x.Select(v => v.Value).ToList())
    .ToList();
    }
    }

  5. I would suggest to use this extension method to chunk the source list to the sub-lists by specified chunk size:

    using System.Collections.Generic;
    using System.Linq;

    ///
    /// Helper methods for the lists.
    ///
    public static class ListExtensions
    {
    public static List<List> ChunkBy(this List source, int chunkSize)
    {
    return source
    .Select((x, i) => new { Index = i, Value = x })
    .GroupBy(x => x.Index / chunkSize)
    .Select(x => x.Select(v => v.Value).ToList())
    .ToList();
    }
    }

Leave a Reply

Your email address will not be published. Required fields are marked *