Azure Series - Cosmos DB - Understanding Consistency

Cosmos DB is a distributed NoSQL database offered by Microsoft Azure, designed to handle large-scale applications with global reach. One of the essential features of Cosmos DB is its consistency model. In this article, we'll explore the concept of Cosmos DB consistency in simple terms, highlighting its significance in data availability and performance.

What is Consistency in Cosmos DB?

In the context of Cosmos DB, consistency refers to the level of data synchronization across different replicas of the database in a distributed environment. When multiple copies of data are spread across data centers globally, ensuring consistency becomes crucial to maintain data accuracy and reliability.

Types of Consistency Levels:

Cosmos DB offers five different consistency levels, each providing varying trade-offs between data consistency and performance. Let's explore them in simple terms:

1. Strong Consistency:

   Strong consistency ensures that all replicas of data are synchronized at all times. When a write operation is performed, all subsequent read requests will return the latest data. While this provides the highest level of data accuracy, it might impact performance due to increased latency for read operations.

2. Bounded Staleness:

   Bounded staleness guarantees that read operations will return data that is not older than a specified time limit (staleness window). This level offers a balance between strong consistency and performance, allowing some delay in data synchronization but ensuring data is reasonably up-to-date.

3. Session Consistency:

   Session consistency guarantees that read operations performed within the same session will always return the latest data. It is useful for applications where users interact with the same data repeatedly and expect real-time updates without affecting other users' read requests.

4. Consistent Prefix:

   Consistent prefix ensures that read operations will return data in the order in which they were written. This consistency level is suitable for applications that rely on chronological data ordering but may tolerate some inconsistency among different data replicas.

5. Eventual Consistency:

   Eventual consistency allows for temporary inconsistency across replicas, as data might take some time to propagate across the distributed database. This level offers the best performance but can lead to stale data being read during short windows of time.

Choosing the Right Consistency Level:

Selecting the appropriate consistency level depends on your application's requirements. If your application demands real-time, up-to-date data at all times, strong consistency might be suitable. However, if performance is a top priority and eventual consistency is acceptable, you can opt for that.

Conclusion:

Cosmos DB consistency plays a crucial role in ensuring data accuracy and performance for distributed applications. By understanding the various consistency levels and their trade-offs, developers can make informed decisions when designing and deploying their applications on Cosmos DB. Consider your application's needs and user expectations to choose the most appropriate consistency level, striking the right balance between data synchronization and performance.

The provider for the source IQueryable doesn't implement IAsyncQueryProvider (How to unit test code that uses AutoMapper ProjectTo?)

Whenever we like to develop using test-driven development approach, Unit Testing is always essential. When it comes to .Net Core and .Net Entity Framework and Automapper it little bit different, no matter what unit testing framework we use, we need to mock libraries. Also when you use third party packages such as AutoMapper it becomes more difficult and comes with lot of other issues in Unit Testing. Here is one of the issue with automapper function “ProjectTo” I am discussing now.

In my previous article, I explained how to make mocking Asynchronous database calls in Unit Testing  

Ref: http://sandeeppawar.blogspot.com/2020/05/mocking-asynchronous-database-calls-in.html

“ProjectTo” function of Automapper Issue in Unit Testing:

So far it was good, We made decision to use the AutoMapper to map our database entities to Business entities. As plan of this we changed our existing function to use automapper.

Note: I assume the reader of this article is aware of how AutoMapper works.

From my previous article example, we changed our function to use automapper. We used TrainingTypeDto as our business entity. This business entity is already mapped in AutoMapper Profile. E.g.    CreateMap().ReverseMap();

Here is our modified function: _mapper is instance of Automapper.

public async Task<string> GetTrainingTypeDescription(string code)         {                 var dto = _mapper.ProjectTo(_dbContext.Set().Where(x => x.TrainingTypeCode.Equals(code)));                 var result = await dto.FirstOrDefaultAsync(cancellationToken);             if (trainingType != null)                 return trainingType.Descriptions.ToString();             return string.Empty;         }

But we had the same error as below which we were getting while making asynchronous DB calls. After deep dive into the issue we found that the issue was with ProjectTo function of automapper.

System.InvalidOperationException: The provider for the source IQueryable doesn't implement IDbAsyncQueryProvider. Only providers that implement IDbAsyncQueryProvider can be used for Entity Framework asynchronous operations. For more details see http://go.microsoft.com/fwlink/?LinkId=287068.

Possible Reason for failure:

If we want to test this function with in-memory set of data, the test will fail because our ProjectTo function of automapper tries to generate the TrainingTypeDto object which won’t support the interfaces needed to make the asynchronous call. The reason behind this failure is ProjectTo function failed to create the new object which implements the IAsyncQueryProvider interface needed for the entity framework asynchronous extension method.

We already saw how to implement this IAsyncQueryProvider interface. We need to make few changes so that ProjectTo function to return the proper object which can make the help us to mock the asynchronous calls. See the changes are highlighted in Yellow Background in your previously created helping classes.

internal class TestAsyncEnumerable<T> : EnumerableQuery, IAsyncEnumerable, IQueryable     {         public TestAsyncEnumerable(IEnumerable enumerable)             : base(enumerable)         { }         public TestAsyncEnumerable(Expression expression)             : base(expression)         { }         public IAsyncEnumerator GetAsyncEnumerator(CancellationToken cancellationToken)         {             return GetEnumerator();         }         public IAsyncEnumerator GetEnumerator()         {             return new TestAsyncEnumerator(this.AsEnumerable().GetEnumerator());         }         IQueryProvider IQueryable.Provider         {             get { return new TestAsyncQueryProvider(this); }         }     } internal class TestAsyncEnumerator<T> : IAsyncEnumerator     {         private readonly IEnumerator _inner;         public TestAsyncEnumerator(IEnumerator inner)         {             _inner = inner;         }         public ValueTask DisposeAsync()         {             _inner.Dispose();             return new ValueTask();         }         public T Current => _inner.Current;         public ValueTask<bool> MoveNextAsync()         {             return new ValueTask<bool>(_inner.MoveNext());         }     } internal class TestAsyncQueryProvider<TEntity> : IAsyncQueryProvider     {         private readonly IQueryProvider _inner;         internal TestAsyncQueryProvider(IQueryProvider inner)         {             _inner = inner;         }         public IQueryable CreateQuery(Expression expression)         {             switch (expression)             {                 case MethodCallExpression m:                     {                         var resultType = m.Method.ReturnType; // Should be IQueryable                         var tElement = resultType.GetGenericArguments()[0];                         var queryType = typeof(TestAsyncEnumerable<>).MakeGenericType(tElement);                         return (IQueryable)Activator.CreateInstance(queryType, expression);                     }             }             return new TestAsyncEnumerable(expression);         }         public IQueryable CreateQuery<TElement>(Expression expression)         {             return new TestAsyncEnumerable(expression);         }         public object Execute(Expression expression)         {             return _inner.Execute(expression);         }         public TResult Execute<TResult>(Expression expression)         {             return _inner.Execute(expression);         }         public IAsyncEnumerable ExecuteAsync<TResult>(Expression expression)         {             return new TestAsyncEnumerable(expression);         }         public TResult ExecuteAsync<TResult>(Expression expression, CancellationToken cancellationToken)         {             var expectedResultType = typeof(TResult).GetGenericArguments()[0];             var executionResult = typeof(IQueryProvider)                 .GetMethod(                     name: nameof(IQueryProvider.Execute),                     genericParameterCount: 1,                     types: new[] { typeof(Expression) })                 ?.MakeGenericMethod(expectedResultType)                 .Invoke(this, new[] { expression });             return (TResult)typeof(Task).GetMethod(nameof(Task.FromResult))                 ?.MakeGenericMethod(expectedResultType)                 .Invoke(null, new[] { executionResult });         }     }

There will be no change in our Helper class which generate our mock data.

public class UnitTestHelper     {         public static Mock> GetQueryableMockDbSet<T>(IQueryable testData) where T : class         {             var mockSet = new Mock>();             mockSet.As>()                 .Setup(x => x.GetAsyncEnumerator(default))                 .Returns(new TestAsyncEnumerator(testData.GetEnumerator()));             mockSet.As>()                 .Setup(x => x.Provider)                 .Returns(new TestAsyncQueryProvider(testData.Provider));             mockSet.As>()                 .Setup(x => x.Expression)                 .Returns(testData.Expression);             mockSet.As>()                 .Setup(x => x.ElementType)                 .Returns(testData.ElementType);             mockSet.As>()                 .Setup(x => x.GetEnumerator())                 .Returns(testData.GetEnumerator());             mockSet.As>()                 .Setup(x => x.GetEnumerator())                 .Returns(testData.GetEnumerator());             return mockSet;         }     }

 Next Step is to create the data (No change):

private static IQueryable GetTrainingTypes()         {             return new List             {                 new TrainingType()                 {                     TrainingTypeCode = "Qa",                     Descriptions = " Quality Assurance Training",                     Name =  "QA Training"                 },                 new TrainingType()                 {                     TrainingTypeCode = "Lab",                     Descriptions = "Laboratory Training.",                     Name =  "Labs Training."                 }             }.AsQueryable();         }

Now that we have the test data and the helper classes needed for that data to be accessed asynchronously by Entity Framework, Lets write our test function.

So given our function GetTrainingTypeDescription(), we want to test it to make sure it works and it’s returning the correct value.

[Fact] public async void When_querying_for_a_TrainingTypeDescription__returns_Description()         {             var context = new Mock();             var trainingtype = GetTrainingTypes();             var cancellationToken = new CancellationToken();             context.SetupGet(c => c.TrainingType)                 .Returns(UnitTestHelper.GetQueryableMockDbSet( trainingtype).Object); context.SetupSet(c => c.TrainingType)                 .Returns(UnitTestHelper.GetQueryableMockDbSet( trainingtype).Object);             // Call the function to test     var typeHandler = new TypeHandler(context.Object);     var result = await TypeHandler.GetTrainingTypeDescription("Lab");             Assert.True(result.HasValue);         }

Now, Run the Test. It will be successfully completed without errors. You can debug and see how it works.