Splitting a Generic List<T> into Multiple Chunks
Greetings visitor from the year 2020! You can get the latest code for this from my Github repo here. Thanks for visiting.
“Chunking” is the technique used to break large amount of work into smaller and manageable parts. Here are a few reasons I can think of why you want to chunk, especially in a batch process where you have to process large number of items:
- Manage/minimize peak memory requirement.
- During failures, the entire process can resume at the last failure point, instead of all the way from the beginning.
- Take advantage of multiple processors/cores (by having multiple threads, each processing a small chunk).
Here’s a helper method to quickly split a List<T> into chunks:
/// <summary> /// Splits a <see cref="List{T}"/> into multiple chunks. /// </summary> /// <typeparam name="T"></typeparam> /// <param name="list">The list to be chunked.</param> /// <param name="chunkSize">The size of each chunk.</param> /// <returns>A list of chunks.</returns> public static List<List<T>> SplitIntoChunks<T>(List<T> list, int chunkSize) { if (chunkSize <= 0) { throw new ArgumentException("chunkSize must be greater than 0."); } List<List<T>> retVal = new List<List<T>>(); int index = 0; while (index < list.Count) { int count = list.Count - index > chunkSize ? chunkSize : list.Count - index; retVal.Add(list.GetRange(index, count)); index += chunkSize; } return retVal; }
If you want to be more efficient at the cost of readability, the second version below moves the items from the big list into the small chunks, so both types of lists will not need to be in memory at once:
/// <summary> /// Break a <see cref="List{T}"/> into multiple chunks. The <paramref name="list="/> is cleared out and the items are moved /// into the returned chunks. /// </summary> /// <typeparam name="T"></typeparam> /// <param name="list">The list to be chunked.</param> /// <param name="chunkSize">The size of each chunk.</param> /// <returns>A list of chunks.</returns> public static List<List<T>> BreakIntoChunks<T>(List<T> list, int chunkSize) { if (chunkSize <= 0) { throw new ArgumentException("chunkSize must be greater than 0."); } List<List<T>> retVal = new List<List<T>>(); while (list.Count > 0) { int count = list.Count > chunkSize ? chunkSize : list.Count; retVal.Add(list.GetRange(0, count)); list.RemoveRange(0, count); } return retVal; }
