Migrate your asp.net settings to Azure App Configuration

Before asp.net core, we only had the web.config file to configure and read the application settings from code. The framework didn’t have a plugable way of adding an extra configuration providers. Any other confiruation data sources meant a custom solution and custom code to read and update configuration keys.

With asp.net core, we got the new appsettings.json as one source of configuration. Along with it came the environment variables, other files, secret keys that can be used to omit connection strings and passwords from being stored on source control.

Things event got better with Azure App Service that has a settings/configuration section that can override the application settings and connection strings sections. This applies to both asp.net framework and asp.net core.

Isn’t this enough already?

It is good as a start. but, it has a major issue. If you change any value in your configuration keys, it means the application must be restarted which affects the logged in users and may cause data loss.
Another issue is that our application code and configuration exists in the same place which contradicts with the twelve factor . These configuration keys cannot also be shared with other applications except using copy/paste

Azure App Configuration

Azure App Configuration is a service that is still in preview that can be a central place for all your configuration keys and feature flags.
It is so easy to add it to your asp.net core configuration provider using the provided nuget packages.

Add the need packages

First, install the package Microsoft.Azure.AppConfiguration.AspNetCore . You may need to check that you enabled prerelease versions if you are using visual studio

In your Program.cs file, add the following code which will inject Azure app configuration as a configuration provider. Notice that you will have to add a key named “AppConfig” in your connectionStrings section in appsettings.json that will store the connection string for your Azure App Configuration

public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
    WebHost.CreateDefaultBuilder(args)
        .ConfigureAppConfiguration((hostingContext, config) =>
        {
            var settings = config.Build();
            config.AddAzureAppConfiguration(settings["ConnectionStrings:AppConfig"]);
        })
        .UseStartup<Startup&gt;();

Now, your application is ready to read any configuration keys from Azure App Configuration. All you need to do is to create a new instance by following this link

Reuse your existing configuration keys

It’s not only too easy to add Azure App Configuration to your asp.net core app but also migrating your existing keys. You don’t want to end up with a situation where you have to spend hours moving your keys manually or writing scripts to do it. Azure App Configuration can do that for you.

As you can see above, it is too simple. Just choose Import/export from the left menu then choose where you want App Configuration to import the values from:

  1. App Configuration: you can import the values from another Azure App Configuration instance
  2. App Service: which can be perfect in our case since it will import the values from your Azure App Service
  3. Configuration File: you can pass the appsettings.json directly and it will import the values from there

Once you did that, you will have all keys/connection strings imported and ready to go.

Conclusion

Azure App Configuration is a great service that still in preview but it enables you to offload your application from configuration keys and connection strings. It also has the auto refresh which will allow your app to always have the latest config keys without the need to restart your application and impact your users.
It also has feature management which is a great addition but we will talk about it in future posts.

Advertisements

Understanding Azure Bot State

When you chat with a bot built with Azure Bot Service, you will have to maintain user state. Example, the options that the user chose last time, or the usernames of all users in a conversation.

Conversation State Types

Azure Bot Service has 3 different states:

User State
This state is related to the user on a specific channel regardless how many conversation they have. So, if the user is chatting with a bot from Facebook channel and connected to the same bot in another conversation , any values saved in user state will be visible in both chats. This is useful in scenarios where you want to save an info about the user like the username or preference that will be used in all conversation for this user.

Conversation State
This state is scoped to the conversation only but not to each user. If you have 3 users chatting with the bot, any data saved in conversation state will be shared across all users. This is used in cases where the data saved is not related to a specific user such as statistics. If the user connected to another session with the bot, the info saved her will not be visible in the other conversation

Private Conversation State
Here, the framework manages the state for each user in each conversation. Imagine a scenario where the user is submitting a ticket. You want the ticket info to be scoped to this conversation only because they may be submitting another ticket with another conversation.

The state storage saves each data in a key/value store. The keys used for each state type is as below:

  • User Sate: {Activity.ChannelId}/users/{Activity.From.Id}#YourPropertyName
  • Conversation state: {Activity.ChannelId}/conversations/{Activity.Conversation.Id}#YourPropertyName
  • Private conversation: {Activity.ChannelId}/conversations/{Activity.Conversation.Id}/users/{Activity.From.Id}#YourPropertyName

Conversation Storage

By default, the state variables are stored in memory. But you can change that from the startup.cs. In the example below, I have configured it to use Azure Table storage to store the state, this makes it persistent. You can create your own state by inheriting from IStorage

Next, you configure which state types you want to use, in this example, I use the conversation state. To use the user state, you have to create an instance of UserState and add it as a singleton just like conversation state.

public void ConfigureServices(IServiceCollection services)
        {
          //IStorage dataStore = new MemoryStorage(); 
          IStorage dataStore = new Microsoft.Bot.Builder.Azure.AzureBlobStorage("connection string", storageContainer);

          var conversationState = new ConversationState(dataStore);
            services.AddSingleton(conversationState);
        }

State property accessors

https://docs.microsoft.com/en-us/azure/bot-service/v4sdk/media/bot-builder-state.png?view=azure-bot-service-4.0

Now, its time to use the state in our bot. To do that, you have to use a property accessor which allows you to read and update data to the state storage. Note that calling the GetAsync and SetAsync on the state accessor doesn’t actually save the state data in the storage system. You have to call the SaveChangesAsync as you can see from the sequence diagram above.

To understand more about state property accessors and how to use it to read and update data into bot state, go through this link

Azure Bot Service Overview

Today, I started learning Azure BOT Service, I will summarise each section into a blog post.

What is BOT?

A BOT is a modern way of user interaction. Instead of user clicking buttons and typing data in text fields, the user will be chatting or talking to an application that will understand the human language and respond accordingly.
This of course increase the human interaction with your site or application and also decrease the need for employees to handle user requests.
Imagine a user who wants to know about car insurance quote, instead of calling the insurance company, they can chat with the bot or even call it and ask them about the prices and the bot can respond as if it was a human or delegate the call to a human if it is not able to assist. This can save a lot for the company.

How BOT works?

Users interact with the BOT through a channel. A channel can be DirectLine like the case when you chat directly with the BOT from web site. It can be Facebook Messenger, Twitter, or any other popular social media platform.
The chat between users and BOT can be a text messages, a card that display graphic content to the user or a voice message or even an interactive form that ask the user to enter some information like SSN or Policy Number.
BOT can also use AI services such as Azure Congnitive Services to make it smarter.

https://haithamshaddad.files.wordpress.com/2019/02/84064-1xmuakcv_xkoszd4odvrika.png

BOT HTTP Communication

The communication between the BOT and Users/Channels take place through HTTP Post requests. When a user joins a conversation, an HTTP message is posted to the BOT service and the BOT just acknowledge the message with 200 OK response. Then, the BOT joins the conversation and again another HTTP Post message is sent to the channel and the channel replies back with a 200 Ok. These messages are called ConversationUpdate messages.
All these messages between the BOT and the channel are called activities. During the ConversationUpdate activity, you can check the members added or removed.
Once the connection is established, the messages send from the channel to the BOT are sent as an HTTP Post requests and the BOT replies back with a different HTTP Post not via response to the original HTTP Post request. So, as you can see from the image below, the Channel sent a Hi message, the BOT just echoed the message but it can send a different message. Then, the channel confirmed with a 200 Ok to the BOT and finally the BOT replies back to the original hi message with 200 Ok.
With the current version (4) it may not be very clear about the HTTP post messages but if you tried with the older version (3), it was very clear and when you create a new project you can clearly see an API controller called MessagesController and you can access it via /api/messages.

activity diagram
https://docs.microsoft.com/en-us/azure/bot-service/v4sdk/media/bot-builder-activity.png

Defining Rounds

Humans talk one at a time, one talk and the others listen (At least this is what should happen ūüôā ).

activity processing stack
https://docs.microsoft.com/en-us/azure/bot-service/v4sdk/media/bot-builder-activity-processing-stack.png

The important thing to note in the above image is the TurnContext
which holds information about whose turn and what data was sent (Typically JSON) and the middleware which holds some components that execute in order to do something with the activity received. After all middleware components finish execution, it passes the control to the Turn Handler which will execute the application logic that will decide the response that will be sent to the channel/user.
The middleware is executed on the incoming activity received from the channel and also on the outbound activity sent from the bot to the channel itself.

Azure Functions anatomy – Part 2 – What is inside a Function App?

In the previous post (Azure Functions 2.0 Anatomy- Part 1), we saw the structure of a function app. In this part, I will explain the different components of an Azure Function.

Components

Every Azure Function 2.0 consists of the following components:

  • Trigger
  • Bindings (one of which is the trigger). Binding can be input or output
  • Value Converter/Binder
  • Listener

A binding can be a Blob file, a queue message, an event from event hub or event grid or any custom binding developed by anyone. One of these bindings can initiate the function call and in this case it is called a Trigger.
A binding can also be Input or Output. Input binding delivers data to the function and output binding is used by the function to write data.

Now, assume we have the following function and its configuration

        [FunctionName("SampleFunction")]
        public static void Run([QueueTrigger("myqueue-items", Connection = "AzureWebJobsStorage")]string myQueueItem, [Blob("sample-output")]Stream output, TraceWriter log)
        {
            log.Info($"C# Queue trigger function processed: {myQueueItem}");
        }

This function has 2 binding parameters. myQueueItem which is both an input binding and a trigger, and output which is an output binding.

How Azure Functions call this method and convert the queue message to a string and the stream to a Blob file?

Binding, Listener, Value Provider and Converter

When the azure function app starts, it scans all functions that exist in the app directory like we saw in the previous post, and for each function it reads the input, output and trigger from the function config JSON file.

The scanning process uses a descriptor provider which creates the input binding, output binding, trigger and an invoker

Once all functions are loaded, it tried to load the types for each function that was developed outside the portal. It uses the scriptFile and EntryPoint as you can see below. The entry point in the previous code snippet is FunctionApp1.Function1.Run method that exist in the assembly FunctionApp1.dll

{
  "generatedBy": "Microsoft.NET.Sdk.Functions-1.0.14",
  "configurationSource": "attributes",
  "bindings": [
    {
      "type": "timerTrigger",
      "schedule": "0 */5 * * * *",
      "useMonitor": true,
      "runOnStartup": false,
      "name": "myTimer"
    }
  ],
  "disabled": false,
  "scriptFile": "../bin/FunctionApp1.dll",
  "entryPoint": "FunctionApp1.Function2.Run"
}

Each function app can either have a DLL function or a function created by portal. But not both.

Now, lets see what are the other components inside a binding:

  • Listener (IListener): This one has a method named StartAsync, which basically start listening for an event to trigger the function. For Azure Blob trigger, it scans the logs for new or modified files. For Queue trigger, it check for new messages in the specified queue.
  • Value Provider (IValueProvider): Once the listener has the event that should trigger the function, it uses the value provider to fill all input parameters. If the trigger was a blog trigger, then the value will be the file, the metadata, the name and content of the file itself. For a Queue, it will be content of the message and so on and so forth.
  • Value Converter (IAsyncConverter): This will be used to convert the values found in the value provider to all the input parameters configured for the function. In the previous example, StorageQueueMessageToStringConverter will be used to convert the CloudQueueMessage instance to the string parameter named myQueueItem. The same will be used to write data to output parameters, the framework will choose the best converter that matches the Output parameter type (Stream) and the type of variable that will be used to write to stream, ex: String or any Stream Writer.
  • Executor (ITriggerExecutor): Now we know that the function should be triggered, we have the data from input binding and we bound all input parameters, now we need to call the function body. This takes place using an implementation of ITriggerExecutor which can use reflection to call the function or in a very special case, it may get executed as a web hook like the case with HTTP trigger.

In the next post, I will dig more into Azure Function App to explore the extensions and host.

Azure Functions 2.0 anatomy – Part 1

In these series of posts, I will explain the internals of Azure Functions and how it works. By the end of it, you should be able to understand what happens once an Azure Function is deployed till the moment it is triggered and invocation completes.

Is it a Web App?

Yes, every Azure Function App that uses .Net as a Language runs as as.net core web app. When the app starts, it loads all your functions and proxies and set up the routes and listeners for each function.

Web App Structure

Just like any other Azure Web App, it consists of the following folders:

  • data: This folder contains another folder named “functions” which has another folder called extensions. The extensions folder has a list of all extensions installed to your Functions App. Each file has a reference to a nuget package that has the extension implementation.
  • Log Files
  • site: it has the wwwroot which hosts the asp.net core web app and all the functions
Data -> Functions folder structure

But for Azure Functions App, the following folders are added:

.nuget

Contains all nuget packages installed when you install a new extension. A reference for each extension installed exist in a file under: data\functions\extensions. The sample extension file below specify the package :Microsoft.Azure.WebJobs.Extensions.ServiceBus

In Azure Functions 1.0, it was running on .Net 4.6 and the runtime already included all the supported triggers and bindings. But, with version 2.0, the runtime only includes http and timer binding and all the other bindings can be developed and installed separately using extensions. This allows you to develop your own extensions and upload it without the need to wait for a new version of Azure Functions.

{
  "Id": "d3a33dd0-3941-4acf-a303-72773995901d",
  "Status": 1,
  "StartTime": "2019-01-02T02:49:23.5568341+00:00",
  "EndTime": "2019-01-02T02:50:50.1029788+00:00",
  "Error": null,
  "Properties": {
    "id": "Microsoft.Azure.WebJobs.Extensions.ServiceBus",
    "version": "3.0.0"
  }
}

ASP.NET

This folder contains all data protection keys used in the web app

To have a look at this folder structure and explore it yourself, use the following URL: https://%5BFUNCTION_APP_NAME%5D.scm.azurewebsites.net/DebugConsole

wwwroot folder structure

wwwroot folder structure

As you can see, there are 3 folders. Each folder represent a single function. All DLLs for each function exist in the bin folder.

The host.json file by default only has the version number for the function

{
  "version": "2.0"
}

Inside each function, there is a function.json file that has the configuration for this function.

{
  "generatedBy": "Microsoft.NET.Sdk.Functions-1.0.14",
  "configurationSource": "attributes",
  "bindings": [
    {
      "type": "timerTrigger",
      "schedule": "0 */5 * * * *",
      "useMonitor": true,
      "runOnStartup": false,
      "name": "myTimer"
    }
  ],
  "disabled": false,
  "scriptFile": "../bin/FunctionApp1.dll",
  "entryPoint": "FunctionApp1.Function2.Run"
}

The Azure Functions run-time use this file to load the metadata for each function during application startup. The most important properties are the scriptFile which points to the DLL that has the code and the entryPoint that has the exact c# method name that will be executed when the function triggers.
The configurationSource specifies where to read the function configuration such as connection strings for the blob storage or service bus. attributes means it will be retrieved from code.
The binding key lists all bindings for this function. For this sample, it is a timer and it is the trigger, so it has a single binding. For others, it can be bindings that represents input, output and a trigger.

In the next post, I will explain the internals of each function, and how the bindings and triggers works.

How Azure Functions Blob Trigger works

Introduction

Azure functions enable you to quickly build a piece of functionality that can be triggered by an external system such as Azure Blob Storage, Storage Queue, CosmoDB, Event Hub and the list goes on. In this post, I will explain how Azure Blob Trigger works.

Sample Function

If you created a new Azure Function using Visual Studio, you will end up with the following code:

 public static class Function1
    {
        [FunctionName("Function1")]
        public static void Run([BlobTrigger("samples-workitems/{name}", Connection = "")]Stream myBlob, string name, TraceWriter log)
        {
            log.Info($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
        }
    }

The secret lies in the attribute [BlobTrigger]. It accepts the path to the storage container which Azure Web Jobs will monitor. As you know, Azure Functions is based on Azure Web Jobs, so the same triggers are used.

How Azure Web Job knows about a file that is being added, updated or removed to a container?

Run the sample app you just created in visual studio, it should work on the local storage emulator. Now, you should be able to invoke the function by creating a blob container called samples-workitems in your development storage account and you should end up with the following structure.

blob container

You can see the container “samples-workitems” which Azure Web Jobs will monitor and invoke the function whenever a file is changed there. But there is also another container named “azure-webjobs-hosts” which is the secret for how Azure monitor the files in that container.

If you opened that container, you will find a folder called “blobreceipts” which has a folder for the function name.

Inside the folder with the function name, there are some other folders with strange names like below

etags

These strange names are the ETag of each blob file added, edited or removed. So, when you change a file in any container, Web Jobs will create a new folder here with the new version ETag for that file and inside this folder you will find the same structure of the file being edited. In our case, there should be a folder called samples-workitems and inside this folder the file that was modified.

When a new file is added, Azure will check if  its ETag exist in the azure-webjobs-hosts folder and if not, then it will call the Azure Function. This way it will prevent duplicate calls for the same file. This pattern is called blob receipt.

Note that this process depends on Azure Blob Storage Logging which can take up to 10 minutes to write to the container azure-webjobs-hosts to improve performance. If you need your function to trigger faster then consider using Storage Queue trigger instead.

 

How to build a web application on Azure?

Microsoft Azure is one of the leading cloud platforms that enables us to build a scalable and highly performing web applications. In this article,  I will list the available options that allow you to quickly understand what you need in order to build your application and what services to use in each layer.

What do we need to build a web application?

Each web application needs some services in order to work propably. For example, a web server, a database server, queuing system etc..

In the past few years, I have found many people using Azure wrong or inefficiently. Some used virtual machines while they should have used Web Apps, others used a huge size of a virtual machine because they didn’t know they could scale their VM without losing the data. Accordingly, I will do my best to give you a summery of Azure services that you can utilize to build your next fantastic Web App.

Application Life-cycle Management

So, first things first. You need a place to store your use cases, test cases, source code, run unit tests and perform a continuous integration and deployment.

Although it may not be a part from Azure, but Team Services integrates seamlessly with Azure and many Azure Services like Web Apps, Mobile Apps, Azure Functions … support continuous integration with Azure.

Team Services help you to have a central repository for your source code, documents and requirement. It also supports Visual Studio Team Services and Git as a source control. It is free and supports unlimited number of projects with 5 free users.

Hosting

Now, you developed your web application, tested it and it is time to host it. Azure supports hosting for .Net, Java, PHP, NodeJS, and Python built applications.

In this phase you have more than one option:

  1. Virtual Machine: This is the very basic option and should not be used unless you have no other way, ex: migrating a legacy application that cannot work with a PAAS offering, in this case you simply have a virtual machine in the cloud and you can remotely connect to it, install the needed software and deploy your application. Note that this is the most expensive solution.
  2. Cloud Services: This is the same as Virtual Machine except that it offloads some of the work to be done by you such as setting up software and windows updates. It has 2 types, a web role which is setup to host a web application directly, and a worker role which enables you to host an executable application such as a windows service or a console app. The good part about cloud services is that you provide it with a package that has your code and it manages the deployment of the application. If the virtual machine hosting the app went down, Azure will automatically creates another virtual machine and deploy the code to it.
  3. App Service: This is the Platform as a Service or PAAS offering from Azure, all you have to worry about here is your code. It gives you a vitrual directory on cloud and all you have to do is to deploy your App to it either through web deploy, FTP or upload a package. It also offers continous integration and monitoring capabilities. This is the most flexible and cheapest option and you can autoscalre it according to the CPU usage, RAM usage and other factors. So basically, you can start small and scale as you need later. Azure App Service has Web Apps to host your application, Mobile App which is a backend as a service to your mobile apps, Logic Apps which gives you the option to build your businss logic and integration between different systems, and finally API Apps which is used to host your REST api web Apps.

Integration

If your application integrates with other systems, then you can use Azure Logic App, Azure Functions, BizTalk Service and Service Bus

Most services gives you a very nice visual designer that allows you to orchestrate your business logic and it offers out of the box seamless integration with many external systems such Office 365, Dynamic CRM or On-premise system through BizTalk Services

Data Storage

Azure offers many services to store your data, all of it are based on Azure Storage Account which is the main storage system for all services in Azure. Each account has some storage and throughput limitation, you can check these limitation from here

  1. If you need to store text files or video files, you can use Page and Block blobs. Each type is suitable for specific file types, you can read more about it from here
  2. Use Azure Media Services if you have to store and stream media files.
  3. Azure table storage is suitable if you need a NoSQL database, it provides a high throughput storing and retrieving entities that have no referential integrity between them.
  4. Azure DocumentDB is a NoSQL document database similar to MongoDB, it is a born in the cloud database that you can use if you need a low latency high performing database
  5. Azure SQL Database is a SQL Server database in the cloud where you need not to worry about SQL Server installation, Backup and Restore or anything else, you just create a database and use it.
  6. Azure File Share is used to replace any legacy File System Share in any legacy application.

 

Now, you should be able to have some basic knowledge about the features you can use to build your next web application.