Recently I encountered a strange problem on a project using Service Bus. We were using message sessions and monitoring the
BrokeredMessage.LockedUntilUtc property to check if we needed to renew the lock (in case the message processing takes longer than the lock duration).
One day, seemingly out of the blue,
LockedUntilUtc started returning
DateTime.MaxValue values which crashed our message handler because our monitoring code couldn’t handle it. We did not recently make any changes to the code, update the version running in production or change anything in the production environment. Even worse, we couldn’t replicate the issue in a new Service Bus namespace. So we were quite convinced that the issue couldn’t be with our code.
As it turns out, the issue was caused by the magical combination of developer error and a slowly rolling out change in Azure Service Bus.
First of all, our code for monitoring the lock duration was using the wrong property. For session scenarios, the lock duration is applied to the session, not to the individual messages. So instead of using the
BrokeredMessage.LockedUntilUtc property, we should have been using the
MessageSession.LockedUntilUtc property. Sessions are mainly used for message ordering, and only one receiver per session is allowed at a time. Therefor it makes sense that the lock duration is applied at the session level.
Secondly, the Service Bus team applied an update where the
BrokeredMessage.LockedUntilUtc property for a session message will always return
LockedUntilUtc would return the session lock duration + one day for session messages.
This change was slowly rolling out to the different scale units, which is why a newly created Service Bus namespace did not (yet) show the issue.
I do think
DateTime.MaxValue is a better value to return for a session message as it’s a better indicator of ‘you shouldn’t use this property in this scenario’ than lock duration + one day.
As a summary, see this example for the session message property values before and after the update:
EnqueuedTimeUtc: 3/29/2018 9:26:44 PM
MessageSession.LockedUntilUtc: 3/29/2018 9:27:52 PM
BrokeredMessage.LockedUntilUtc: 3/30/2018 9:26:53 PM
EnqueuedTimeUtc: 5/18/2018 9:22:15 PM
MessageSession.LockedUntilUtc: 5/18/2018 9:29:16 PM
BrokeredMessage.LockedUntilUtc: 12/31/9999 11:59:59 PM