Serialize Huge amount of Query results

Serialize Huge amount of Query results

Problems

There are two problems in that task. 

First - it's not possible to load all the elements into a memory, because of RAM limitations. 

Second - it's required to implement page by page loading from the CRM.

Solution

To resolve described problems it's required to create a custom IEnumerable implementation. That class wouldn't be a full functional collection, but still can be used to control sequential access to the CRM prefiltered data.

The base of our collection is XrmEnumerator, IEnumerator implementation:

public class XrmEnumerator<T> : IEnumerator<T> where T : Entity
{
    private readonly Queue<T> _collected = new Queue<T>();
    private IOrganizationService _service;
    private QueryExpression _query;
    private string _lastCookie;
    private bool _moreRecords;

    public T Current { get; private set; }
    object IEnumerator.Current => Current;

    public XrmEnumerator(IOrganizationService service, 
                         QueryExpression query)
    {
        _service = service;
        _query = query;
        if (query.PageInfo == null)
            query.PageInfo = new PagingInfo
            {
                Count = 5000,
                PageNumber = 1
            };
        FillThePack();
    }

    private void FillThePack()
    {
        var result = _service.RetrieveMultiple(_query);
        _lastCookie = result.PagingCookie;
        result.Entities.ToList()
              .ForEach(e=>_collected.Enqueue(e.ToEntity<T>()));
        _moreRecords = result.MoreRecords;
    }

    public void Dispose()
    {
        _service = null;
        _query = null;
    }

    public bool MoveNext()
    {
        if(_collected.Count == 0)
        {
            if (!_moreRecords)
            {
                return false;
            }
            _query.PageInfo.PagingCookie = _lastCookie;
            _query.PageInfo.PageNumber++;
            FillThePack();
        }
        Current = _collected.Dequeue();
        return true;
    }

    public void Reset()
    {
        _query.PageInfo.PagingCookie = string.Empty;
        _query.PageInfo.PageNumber = 1;
    }
}

The XrmEnumerator has a query and service objects inside and loads a next portion of data into the internal queue _collected. Values are taken from the _collected till there are some elements, if the queue is empty it takes a next portion of data from the CRM. That way it incapsulates the loading of pages and not use a lot of RAM because entities are not linked to anything in the object itself and could be released by the garbage collector.

The collection uses the XrmEnumerator to get the data:

public class XrmEnumerable<T> : IEnumerable<T> where T : Entity
{
    private readonly XrmEnumerator<T> _enumerator;

    public XrmEnumerable()
    { }
    public XrmEnumerable(IOrganizationService service, 
                         QueryExpression query)
    {
        _enumerator = new XrmEnumerator(service, query);
    }

    public IEnumerator GetEnumerator() => _enumerator;
        
    IEnumerator IEnumerable.GetEnumerator()
    {
        return _enumerator;
    }

    public virtual void Add(T entity)
    {
        // do your code on item deserialization
    }
}

Function Add is required by the serializers to be able to (de)serialize a collection. XrmEnumerable does not stores any data so the result of  the deserialization process will be a broken empty collection, that would fail on a request. That is done by purpose, because storing the huge amount of data from can causes an OutOfMemoryExecption. The correct usage would be to create a child class of the XrmEnumerable and override an Add method e.g.:

public class XrmContactsToConsole : XrmEnumerable
{
    public XrmContactsToConsole() { }
    public XrmContactsToConsole(IOrganizationService service, 
                                QueryExpression query)
           : base(service, query)
    {
    }
    public override void Add(Contact entity)
    {
        Console.WriteLine(entity.FullName);
    }
}

See below the sample of working with that collection:

var enumerable = new XrmContactsToConsole(service, query);
var serializer = new DataContractSerializer(typeof(XrmContactsToConsole));
using (var file = File.OpenWrite("contacts.xml"))
    serializer.WriteObject(file, enumerable);

using (var file = File.OpenRead("contacts.xml"))
    serializer.ReadObject(file);

The result of the execution will be the list of names on the console screen.
Try it for your functionality.