Splitting a Generic List<T> into Multiple Chunks

“Chunking” is the technique used to break large amount of work into smaller and manageable parts. Here are a few reasons I can think of why you want to chunk, especially in a batch process where you have to process large number of items:

  • Manage/minimize peak memory requirement.
  • During failures, the entire process can resume at the last failure point, instead of all the way from the beginning.
  • Take advantage of multiple processors/cores (by having multiple threads, each processing a small chunk).

Here’s a helper method to quickly split a List<T> into chunks:

/// <summary>
/// Splits a <see cref="List{T}"/> into multiple chunks.
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="list">The list to be chunked.</param>
/// <param name="chunkSize">The size of each chunk.</param>
/// <returns>A list of chunks.</returns>
public static List<List<T>> SplitIntoChunks<T>(List<T> list, int chunkSize)
{
    if (chunkSize <= 0)
    {
        throw new ArgumentException("chunkSize must be greater than 0.");
    }

    List<List<T>> retVal = new List<List<T>>();
    int index = 0;
    while (index < list.Count)
    {
        int count = list.Count - index > chunkSize ? chunkSize : list.Count - index;
        retVal.Add(list.GetRange(index, count));

        index += chunkSize;
    }

    return retVal;
}
 

If you want to be more efficient at the cost of readability, the second version below moves the items from the big list into the small chunks, so both types of lists will not need to be in memory at once:

 

/// <summary>
/// Break a <see cref="List{T}"/> into multiple chunks. The <paramref name="list="/> is cleared out and the items are moved
/// into the returned chunks.
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="list">The list to be chunked.</param>
/// <param name="chunkSize">The size of each chunk.</param>
/// <returns>A list of chunks.</returns>
public static List<List<T>> BreakIntoChunks<T>(List<T> list, int chunkSize)
{
    if (chunkSize <= 0)
    {
        throw new ArgumentException("chunkSize must be greater than 0.");
    }

    List<List<T>> retVal = new List<List<T>>();

    while (list.Count > 0)
    {
        int count = list.Count > chunkSize ? chunkSize : list.Count;
        retVal.Add(list.GetRange(0, count));
        list.RemoveRange(0, count);
    }

    return retVal;
}

18 Replies to “Splitting a Generic List<T> into Multiple Chunks”

  1. (list.Count – index) gives the number of remaining elements including the index element. So it doesn’t have to be greater than chunkSize, rather greater than or equal.

    I think this line:

    int count = list.Count – index > chunkSize ? chunkSize : list.Count – index;

    should be:

    int count = list.Count – index >= chunkSize ? chunkSize : list.Count – index;

    Thanks for the post – helped me with a problem I was having.

  2. Hi Anjo: I checked my code again and it does work as expected (I did have a pretty comprehensive unit test for it). However, you have very sharp eyes and your version also works just fine. When list.Count – index == chunkSize, either the left side or right side of the equation will get you the same thing.

    Thanks for the comment. It was a good brain excercise.

    Chinh

  3. ///
    /// Paged ForEach
    ///
    ///
    ///
    ///
    ///
    public static void ForEachChunk(this List source, int count, Action<List> action) {
    int nTotal = source.Count();
    var nPageCount = (int)Math.Ceiling((double)(source.Count()) / (double)count);
    for (int i = 0; i < nPageCount; i++) {
    var l = source.Skip(i * count).Take(count).ToList();
    action(l);
    }
    }

  4. here are 2 alternatives that does various things, in various speeds and in various manners:
    … both are extension methods…

    public static class ext
    {
    // Define other methods and classes here
    ///
    /// Break a into multiple chunks. The is cleared out and the items are moved
    /// into the returned chunks.
    ///
    ///
    /// The list to be chunked.
    /// The size of each chunk.
    /// Remove elements from input (reduce memory use)
    /// A list of chunks.
    public static IEnumerable<List> BreakIntoChunks(this List list, int chunkSize = 10, bool remove = false)
    {
    if (chunkSize = list.Count)
    yield return list;
    else
    {
    if(remove)
    {
    while (list.Count > 0)
    {
    int count = list.Count > chunkSize ? chunkSize : list.Count;
    var ret = list.GetRange(0, count);
    list.RemoveRange(0, count);
    yield return ret;
    }
    }
    else
    {
    int count = 0, max = list.Count – (list.Count % chunkSize);
    do
    {
    yield return list.GetRange(count, chunkSize);
    count += chunkSize;
    } while (count < max);

    if(count < list.Count)
    yield return list.GetRange(count, (list.Count % chunkSize));
    }
    }
    }

    //Quick and dirty…
    public static List<List> Split(this List source, int size)
    {
    if (size new { Index = i, Value = x })
    .GroupBy(x => x.Index / size)
    .Select(x => x.Select(v => v.Value).ToList())
    .ToList();
    }
    }

  5. I would suggest to use this extension method to chunk the source list to the sub-lists by specified chunk size:

    using System.Collections.Generic;
    using System.Linq;

    ///
    /// Helper methods for the lists.
    ///
    public static class ListExtensions
    {
    public static List<List> ChunkBy(this List source, int chunkSize)
    {
    return source
    .Select((x, i) => new { Index = i, Value = x })
    .GroupBy(x => x.Index / chunkSize)
    .Select(x => x.Select(v => v.Value).ToList())
    .ToList();
    }
    }

  6. ARIA will be ranking artists based on streaming results from
    deezer mobile, Rdio, Spotify,
    JB Now and Samsung Music Hub, The Guardian said. Audioboom (AIM: BOOM),
    a fairly new entrant in the field that is centered on spoken word audio, meanwhile,
    has just relaunched its website and i – OS app with new features.
    Stitcher’s content partners include NPR, Wall Street Journal,
    the BBC, and CBC.

  7. Getting popular in organization attracts much better
    revenues this approach is adopted really strongly by us to help our clients in cracking great offers.
    It is the thumb rule of the business right now that
    an improved visibility on Social Media Platforms like Instagram sooner or later strengthens
    the sources of earnings for an organization.

    My website buy active instagram followers (burnt-juicebox.tumblr.com)

  8. A long paragraph is intimidating to today’s reader.
    The fact is that just because you are running advertisements and the phone is ringing with new prospects doesn’t
    necessarily mean you are not missing out on new opportunities and first time customers.

    And if you are trying to break in, trying to get read and sold, trying to get attention ahead
    of the A list writers already in every studio rolodex.

  9. My brother suggested I may like this web site. He was entirely right.
    This put up actually made my day. You can not imagine just how
    so much time I had spent for this information! Thanks!

  10. I used to be recommended this website through my cousin. I
    am no longer certain whether or not this publish is written through him
    as no one else understand such designated approximately my problem.
    You are wonderful! Thanks!

  11. I see you don’t monetize your page, don’t
    waste your traffic, you can earn additional bucks every month
    because you’ve got hi quality content. If you want to know how to make extra bucks, search for:
    Boorfe’s tips best adsense alternative

Leave a Reply

Your email address will not be published. Required fields are marked *