LockedUntilUtc change in Azure Service Bus

Recently I encountered a strange problem on a project using Service Bus. We were using message sessions and monitoring the BrokeredMessage.LockedUntilUtc property to check if we needed to renew the lock (in case the message processing takes longer than the lock duration).

One day, seemingly out of the blue, LockedUntilUtc started returning DateTime.MaxValue values which crashed our message handler because our monitoring code couldn’t handle it. We did not recently make any changes to the code, update the version running in production or change anything in the production environment. Even worse, we couldn’t replicate the issue in a new Service Bus namespace. So we were quite convinced that the issue couldn’t be with our code.

As it turns out, the issue was caused by the magical combination of developer error and a slowly rolling out change in Azure Service Bus.

First of all, our code for monitoring the lock duration was using the wrong property. For session scenarios, the lock duration is applied to the session, not to the individual messages. So instead of using the BrokeredMessage.LockedUntilUtc property, we should have been using the MessageSession.LockedUntilUtc property. Sessions are mainly used for message ordering, and only one receiver per session is allowed at a time. Therefor it makes sense that the lock duration is applied at the session level.

Secondly, the Service Bus team applied an update where the BrokeredMessage.LockedUntilUtc property for a session message will always return DateTime.MaxValue. Previously, LockedUntilUtc would return the session lock duration + one day for session messages.

This change was slowly rolling out to the different scale units, which is why a newly created Service Bus namespace did not (yet) show the issue.

I do think DateTime.MaxValue is a better value to return for a session message as it’s a better indicator of ‘you shouldn’t use this property in this scenario’ than lock duration + one day.

As a summary, see this example for the session message property values before and after the update:

Before update:
EnqueuedTimeUtc: 3/29/2018 9:26:44 PM
MessageSession.LockedUntilUtc: 3/29/2018 9:27:52 PM
BrokeredMessage.LockedUntilUtc: 3/30/2018 9:26:53 PM

After update:
EnqueuedTimeUtc: 5/18/2018 9:22:15 PM
MessageSession.LockedUntilUtc: 5/18/2018 9:29:16 PM
BrokeredMessage.LockedUntilUtc: 12/31/9999 11:59:59 PM


dotnetFlix at Build 2018

I just realized that I don’t post a lot of blog posts considering that the previous post was dotnetFlix at Build 2017 😉 Anyway, can’t believe we’re already up to episode 71 now!

At this years Build we spoke with Paige Bailey (Artificial Intelligence), Dan Rosanova (Azure Messaging) and Mark Fussell (Service Fabric Mesh).

Service Fabric announcements at Build 2017

There were a lot of great Azure Service Fabric announcements at Build this year. Most importantly, you can now run Windows Server Containers on Service Fabric. Linux remains in public preview for now, but will be GA’ing soon in the next release. You can read all about these announcements over at the Azure Service Fabric Team blog.

There were two much smaller features that I especially liked though. Both are changes in the portal experience for creating new clusters.

Continue reading “Service Fabric announcements at Build 2017”

Avoiding data loss when using the Azure Event Hubs Processor Host SDK

I’m currently working on a project where we use Azure Event Hubs to collect telemetry data from connected field devices such as gas and electricity meters. This data gets processed in a variety of ways. We use Azure Stream Analytics to transform and forward the data to various subscribers. For more complex scenario’s where the functionality provided by Stream Analytics isn’t sufficient, we use the Event Processor Host SDK (hosted in a stateless Service Fabric service) to process the events.

Continue reading “Avoiding data loss when using the Azure Event Hubs Processor Host SDK”

DEVintersection Las Vegas 2016

I recently visited the DEVintersection conference in Las Vegas.
It’s a very nice conference with interesting workshops. I attended a two-day TypeScript / Angular2 workshop by John Papa and Dan Wahlin.

While there, Edwin van Wijk and I interviewed Scott Guthrie and Scott Hanselman for the dotnetFlix podcast.
You can view these episodes at the following links:

Tech Days NL 2016

The videos of the two Tech Days NL 2016 talks I did are up on channel9:

Both are in Dutch, and unfortunately there’s no sound after the first couple of slides of the second talk due to some technical difficulties 😦

At the conference I also had the chance to talk to both the opening and closing keynote speakers for the dotnetFlix podcast.
You can view these episodes at the following links:

Creating a Service Fabric Diff Application Package using PowerShell

For a Service Fabric project I’ve been working on, the project team has implemented a release management pipeline in Visual Studio Team Services.
When a commit is pushed to the master branch, the build is run and (if succesful) a release is created and automatically deployed.
The release pipeline basically does two things: 1) make sure the environment is up to date using some ARM templates and 2) upgrade the Service Fabric application to the new version.

Upgrading the application requires that you increment the version numbers in the application and service manifests.
One way to deal with this is simply by incrementing the version number of each service whether or not it has actually been modified.
You can do this by using an auto-generated build number as explained in Colin’s blogpost.
In a lot of situations this works fine because Service Fabric allows for zero downtime deployments.

However, we needed to be sure that some services aren’t unnecessarily upgraded.
Some of the stateful services in the application maintain connections to external hardware devices.
We don’t want to interrupt these each time we need to roll-out a change in any of the other services.

Continue reading “Creating a Service Fabric Diff Application Package using PowerShell”