From Service Fabric to App Service

Service Fabric is a very powerful PaaS that is the foundation for many fundamental services in Azure and beyond. However, there are some concerns with maintenance and the general development experience when Service Fabric is chosen as the framework, because the time spent dealing with the framework might outweigh the time dedicated to app development. For that reason, I decided to migrate our PaaS service from Service Fabric to App Service. This document contains a retrospective of the challenges I encountered during that process.

Why migrate from Service Fabric

From Service Fabric documentation:

Azure Service Fabric is a distributed systems platform that makes it easy to package, deploy, and manage scalable and reliable microservices and containers. Service Fabric also addresses the significant challenges in developing and managing cloud native applications.

ServiceFabricSingleSlide

My Service Fabric learning path

  1. Azure Service Fabric for Developer. Jeffrey Richter on //BUILD/
  2. Jeffrey Richter has video training series of Service Fabric.
  3. Service Fabric book
    ServiceFabricBook

The book is available on learning.oreilly.com. The book is available for MSFT FTEs for free. It is 500 pages long.

Covers lots of topics deeply:

  1. Inside Azure Service Fabric - video series Service Fabric PMs meet different customers. Highly recommend watching especially Cesar Ruiz-Meraz: Azure Event Hubs. After watching, ask yourself: How would you, as an architect, design an EventGrid/ApacheKafka cluster without Service Fabric?

EventHubInsidesVideo

Service Fabric challenges:

  1. General infrastructure complexity - read at least half of the book before making a decision to use it.
  2. Stateful services maintenance - I have not found it in the book but found it in one of the conference videos. Service Fabric Stateful Service is literally a database engine running on Service Fabric nodes. Service Fabric generates a replicable transaction log which must be truncated. To understand how database transaction logs are implemented, read SQL Server Internals book.
  3. Service Fabric Application Model - Service Fabric coupling that doesn’t allow it to run on other platforms
    • Marker interfaces. Problem for dependency injection and violation of .NET Framework Design Guidelines
    • Remoting - coupling to network communication. Better to use HTTP service discovery.
    • Service Fabric Mesh never became mature - Kubernetes won.
  4. Service Fabric SDK
    • SDK didn’t work for weeks, and I had to wait for patches.
    • SDK updates installation was enforced and there was no way to escape (couldn’t pin to a specific SDK version).
    • SDK froze frequently and a PC restart was needed a few times a day.
    • Application Model didn’t allow running without the SDK.
  5. Conclusion:
    • Service Fabric is better suited for super highly scaled Stateful Services.
    • Service Fabric features are only necessary in ~ 5% of design cases.
    • SDK development experience is suboptimal.

Where to go from Service Fabric

  1. Kubernetes is the trend, but no one had enough container experience on our team. We had many small issues even without containers and Kubernetes.
  2. App Service is the simplest PaaS in Azure and the easiest to migrate to (my original thinking 😊). There are a few options for hosting in App Service:
    1. Windows IIS. The oldest and most basic option on App Service is to use ASP.NET Core Razor.
    2. Linux Azure Function without a container with .NET Core.
    3. Windows Container - Hyper-V - a disappointing feature because Server 2019 supports container process isolation, but such a new feature was not available on App Service.
    4. Linux Container - Process isolation - the most lightweight container and the most compatible with Kubernetes.

How to migrate from Service Fabric

  1. Common logic extraction. Move the maximum amount of common logic into .NET Standard assemblies and reference it from two applications (Service Fabric and App Service). CommonLogic
  2. Dependency Injection Injection of Service Fabric dependencies in ASP.NET Core 2+ allows a single application entry point to start with different dependency configurations depending on detection of the Service Fabric environment.

Service Fabric agnostic - Dependency Injection

  1. Read the Dependency Injection book and refactor to Dependency Injection “Dependency Injection Principles, Practices, and Patterns” is the ultimate source of learning about Dependency Injection on .NET. DependencyInjectionBook

Mark Seemann is an ex-Microsoft architect and now he is a cofounder of a consulting company. He has a very respected blog about architecture, dependency injection, and functional programming. \ MarkSeeman ServiceFabricDependencyInjection

  1. Refactor solution logic that can be refactored to dependency injection.
  2. Detect Service Fabric from Program.cs
// Program.cs 
// void Main(string[] args) 

if (!string.IsNullOrWhiteSpace(Environment.GetEnvironmentVariable("Fabric_ApplicationId"))) 
{ 
 // Service Fabric AppModel activation 
 ServiceRuntime.RegisterServiceAsync( 
     "WebFrontendType", 
     context => new WebFrontend(context) 
 ).GetAwaiter().GetResult(); 
 // Prevents this host process from terminating so services keep running. 
 // Looks like Service Fabric process wants to live forever :) 
 Thread.Sleep(Timeout.Infinite); 
} 
else 
{
  var appSettings = new AppSettings(new AppSettingsKeyValueSourceFromEnvironmentVariables()); 
  void AddServices(IServiceCollection services) 
  { 
    // Configure dependency injection composition before Startup is called. 
    services.AddSingleton<AppSettings>(appSettings); 
    services.AddSingleton<IApplicationVersionContext, ExecutableApplicationVersionContext>(); 
    ... 
  }

  var webHostBuilder = WebHost 
      .CreateDefaultBuilder<Startup>(args) 
      .UseKestrel() 
      .ConfigureServices(AddServices); 
  webHostBuilder.Build().Run(); 
}
  1. Clean up ASP.NET Core Startup.cs
    • Startup must be ServiceFabric agnostic.
    • Startup may contain only services that will work with both Service Fabric and self-hosted applications.
    • Use the Options pattern in ASP.NET Core. The Options pattern helps avoid options hardcoding and initialize options using Dependency Injection.
  public sealed class ConfigureHttpsRedirectionOptions : IConfigureOptions<HttpsRedirectionOptions> 
  { 
      private readonly Domain.CommonConfiguration.IAppSettings appSettings; 

      // Dependency injection of settings that may come either from Service Fabric Configuration or from self-hosted app environment variables.
      public ConfigureHttpsRedirectionOptions(Domain.CommonConfiguration.IAppSettings appSettings) 
      { 
          this.appSettings = appSettings; 
      }

      public void Configure(HttpsRedirectionOptions options) 
      { 
          // Initialize Options object. 
          options.RedirectStatusCode = StatusCodes.Status307TemporaryRedirect; 
          options.HttpsPort = appSettings.FrontendPortalHostPortHttps; 
      }
  }

Geneva Monitoring agent on App Service

Geneva Monitoring Agents run in first-party subscriptions as an App Service extension. Read Geneva Docs for Setting Up an Azure App Service to Collect Logs and Metrics.

Certificates on App Service

On Service Fabric we were importing Key Vault certificates to VMSS machine profile using the VMSS Key Vault extension. There are a few ways to work with certificates on App Service:

  1. “Microsoft.Web/certificates” resource.
  2. Construct new X509Certificate2 from imported byte array.
  3. Certificate ephemeral key sets.

“Microsoft.Web/certificates” resource

Certificate may be imported from KeyVault to AppServiceFarm and loaded using User CertificateStorage. By default, IIS hosted Website runs without user Profile. SSL/TLS and AAD authentication certificates may be imported to AppServiceFarm using ARM template. Certificate is kind of a hidden resource type. To see it you should check “Show hidden types” checkbox in Azure portal resource group blade. “Microsoft.Web/certificates”

{
      "type": "Microsoft.Web/certificates", 
      "apiVersion": "2018-11-01", 
      "name": "[variables('customHostNameCertificateName')]", 
      "location": "[resourceGroup().location]", 
      "properties": { 
        "KeyVaultId": "[resourceId('Microsoft.KeyVault/vaults', parameters('keyVaultNameRegional'))]", 
        "KeyVaultSecretName": "[parameters('keyVaultSecretName')]", 
        "password": "", 
        "serverFarmId": "[concat(resourceGroup().id, '/providers/Microsoft.Web/serverfarms/', parameters('serverFarmName'))]" 
      }, 

      "dependsOn": [] 
    },

Imported certificate may be associated with App Service custom domain with ARM “Microsoft.Web/sites/hostNameBindings” resource or using Azure Portal. Imported certificate is autorotated daily.
Certificate may be read by application from User certificate store after settings Website configurations:

{ 
  "name": "WEBSITE_LOAD_CERTIFICATES", 
  "value": "*" 
}, 
{ 
  "name": "WEBSITE_DELAY_CERT_DELETION", 
  "value": "1" 
},

“WEBSITE_LOAD_CERTIFICATES” activates user profile and allows application to read certificates from Certificate User store.

By default, during autorotation new versions of a certificate is installed on AppService and an old version of the certificate is removed immediately. It may cause problem for application that already started and loaded certificate with private key from User Store. Certificate rotation will remove old certificate private key from the disk and will cause a cryptographic failure. To prevent this, use new “WEBSITE_DELAY_CERT_DELETION” setting that track used certificates during application process lifecycle and delays their deletion until process will be restarted on Website restart or redeploy.

Installed certificate can be used for cryptography.

let readCertificateBySubject(subject: string) = 
    use store = new X509Store(StoreName.My, StoreLocation.CurrentUser) 
    store.Open(OpenFlags.ReadOnly); 
    let col = store.Certificates.Find(X509FindType.FindBySubjectName, subject, false)

For C# sample see Load certificate in Windows apps

For container cases see Load certificate in Linux/Windows containers

\

Aur service used some old .NET Framework libraries that were designed 15 years ago before Azure appeared using .NET 3 Windows Identification Foundation.

  1. Certificates
  2. DSTS cert
  3. SSL cert
  4. AAD service identity client cert
  5. In memory certificate
  6. Installed certificate 4.3 - Custom headers WS-Federation 4.4 - One Cert 4.5 - SNAT ports

Construct new X509Certificate2 from imported byte array

By default, IIS hosted Website runs without user Profile and a certificate may be constructed from byte array with MachineKeySet.

new X509Certificate2(Convert.FromBase64String secret, "", X509KeyStorageFlags.MachineKeySet)

This approach is not the best choice for a few reasons:

  1. App Service has a limit (~60 certificates) on the number of certificates that can be created with MachineKeySet. Each X509Certificate2 constructor call persists private key on the disk and limit will be reached very soon.
  2. X509Certificate2 with MachineKeySet must have singleton lifecycle. X509Certificate2 object Dispose/Finalization will remove private key. If two X509Certificate2 are constructed and linked to the same private key, then disposing first one will break the second one and cause CryptographicException inability to use private key in Cryptographic API.

That’s why to create X509Certificate2 it is better to use user profile: 1. Set WEBSITE_LOAD_CERTIFICATES 2. Create X509Certificate2 in user profile with PersistKeySet. PersistKeySet means that certificate key is physically stored on the disk but will not be removed after X509Certificate2 object Disposal/Finalization.

new X509Certificate2(Convert.FromBase64String secret, "", X509KeyStorageFlags.UserKeySet ||| X509KeyStorageFlags.PersistKeySet)

Certificate ephemeral key sets

Construction X509Certificate2 from byte array and store it only in process memory is a simple option but X509KeyStorageFlags.EphemeralKeySet is supported only in .NET Framework or .NET 5. Also, EphemeralKeySet is not supported on MacOS.

Custom headers WS-Federation

We used WS-Federation authentication. AppService has header size limit to prevent DOS attacks. To avoid header overload, we store user groups claims in session. WsFederationOptions.Events.OnTicketReceived allows to customize ClaimsIdentity.

SNAT ports limit

Read Azure AppService HttpClientFactory - use cached but fresh http message handlers

THE END - No it is just the beginning. Next stop is Containers :)