Azure Functions anatomy – Part 2 – What is inside a Function App?

In the previous post (Azure Functions 2.0 Anatomy- Part 1), we saw the structure of a function app. In this part, I will explain the different components of an Azure Function.

Components

Every Azure Function 2.0 consists of the following components:

  • Trigger
  • Bindings (one of which is the trigger). Binding can be input or output
  • Value Converter/Binder
  • Listener

A binding can be a Blob file, a queue message, an event from event hub or event grid or any custom binding developed by anyone. One of these bindings can initiate the function call and in this case it is called a Trigger.
A binding can also be Input or Output. Input binding delivers data to the function and output binding is used by the function to write data.

Now, assume we have the following function and its configuration

        [FunctionName("SampleFunction")]
        public static void Run([QueueTrigger("myqueue-items", Connection = "AzureWebJobsStorage")]string myQueueItem, [Blob("sample-output")]Stream output, TraceWriter log)
        {
            log.Info($"C# Queue trigger function processed: {myQueueItem}");
        }

This function has 2 binding parameters. myQueueItem which is both an input binding and a trigger, and output which is an output binding.

How Azure Functions call this method and convert the queue message to a string and the stream to a Blob file?

Binding, Listener, Value Provider and Converter

When the azure function app starts, it scans all functions that exist in the app directory like we saw in the previous post, and for each function it reads the input, output and trigger from the function config JSON file.

The scanning process uses a descriptor provider which creates the input binding, output binding, trigger and an invoker

Once all functions are loaded, it tried to load the types for each function that was developed outside the portal. It uses the scriptFile and EntryPoint as you can see below. The entry point in the previous code snippet is FunctionApp1.Function1.Run method that exist in the assembly FunctionApp1.dll

{
  "generatedBy": "Microsoft.NET.Sdk.Functions-1.0.14",
  "configurationSource": "attributes",
  "bindings": [
    {
      "type": "timerTrigger",
      "schedule": "0 */5 * * * *",
      "useMonitor": true,
      "runOnStartup": false,
      "name": "myTimer"
    }
  ],
  "disabled": false,
  "scriptFile": "../bin/FunctionApp1.dll",
  "entryPoint": "FunctionApp1.Function2.Run"
}

Each function app can either have a DLL function or a function created by portal. But not both.

Now, lets see what are the other components inside a binding:

  • Listener (IListener): This one has a method named StartAsync, which basically start listening for an event to trigger the function. For Azure Blob trigger, it scans the logs for new or modified files. For Queue trigger, it check for new messages in the specified queue.
  • Value Provider (IValueProvider): Once the listener has the event that should trigger the function, it uses the value provider to fill all input parameters. If the trigger was a blog trigger, then the value will be the file, the metadata, the name and content of the file itself. For a Queue, it will be content of the message and so on and so forth.
  • Value Converter (IAsyncConverter): This will be used to convert the values found in the value provider to all the input parameters configured for the function. In the previous example, StorageQueueMessageToStringConverter will be used to convert the CloudQueueMessage instance to the string parameter named myQueueItem. The same will be used to write data to output parameters, the framework will choose the best converter that matches the Output parameter type (Stream) and the type of variable that will be used to write to stream, ex: String or any Stream Writer.
  • Executor (ITriggerExecutor): Now we know that the function should be triggered, we have the data from input binding and we bound all input parameters, now we need to call the function body. This takes place using an implementation of ITriggerExecutor which can use reflection to call the function or in a very special case, it may get executed as a web hook like the case with HTTP trigger.

In the next post, I will dig more into Azure Function App to explore the extensions and host.

Advertisements

Azure Functions 2.0 anatomy – Part 1

In these series of posts, I will explain the internals of Azure Functions and how it works. By the end of it, you should be able to understand what happens once an Azure Function is deployed till the moment it is triggered and invocation completes.

Is it a Web App?

Yes, every Azure Function App that uses .Net as a Language runs as as.net core web app. When the app starts, it loads all your functions and proxies and set up the routes and listeners for each function.

Web App Structure

Just like any other Azure Web App, it consists of the following folders:

  • data: This folder contains another folder named “functions” which has another folder called extensions. The extensions folder has a list of all extensions installed to your Functions App. Each file has a reference to a nuget package that has the extension implementation.
  • Log Files
  • site: it has the wwwroot which hosts the asp.net core web app and all the functions
Data -> Functions folder structure

But for Azure Functions App, the following folders are added:

.nuget

Contains all nuget packages installed when you install a new extension. A reference for each extension installed exist in a file under: data\functions\extensions. The sample extension file below specify the package :Microsoft.Azure.WebJobs.Extensions.ServiceBus

In Azure Functions 1.0, it was running on .Net 4.6 and the runtime already included all the supported triggers and bindings. But, with version 2.0, the runtime only includes http and timer binding and all the other bindings can be developed and installed separately using extensions. This allows you to develop your own extensions and upload it without the need to wait for a new version of Azure Functions.

{
  "Id": "d3a33dd0-3941-4acf-a303-72773995901d",
  "Status": 1,
  "StartTime": "2019-01-02T02:49:23.5568341+00:00",
  "EndTime": "2019-01-02T02:50:50.1029788+00:00",
  "Error": null,
  "Properties": {
    "id": "Microsoft.Azure.WebJobs.Extensions.ServiceBus",
    "version": "3.0.0"
  }
}

ASP.NET

This folder contains all data protection keys used in the web app

To have a look at this folder structure and explore it yourself, use the following URL: https://%5BFUNCTION_APP_NAME%5D.scm.azurewebsites.net/DebugConsole

wwwroot folder structure

wwwroot folder structure

As you can see, there are 3 folders. Each folder represent a single function. All DLLs for each function exist in the bin folder.

The host.json file by default only has the version number for the function

{
  "version": "2.0"
}

Inside each function, there is a function.json file that has the configuration for this function.

{
  "generatedBy": "Microsoft.NET.Sdk.Functions-1.0.14",
  "configurationSource": "attributes",
  "bindings": [
    {
      "type": "timerTrigger",
      "schedule": "0 */5 * * * *",
      "useMonitor": true,
      "runOnStartup": false,
      "name": "myTimer"
    }
  ],
  "disabled": false,
  "scriptFile": "../bin/FunctionApp1.dll",
  "entryPoint": "FunctionApp1.Function2.Run"
}

The Azure Functions run-time use this file to load the metadata for each function during application startup. The most important properties are the scriptFile which points to the DLL that has the code and the entryPoint that has the exact c# method name that will be executed when the function triggers.
The configurationSource specifies where to read the function configuration such as connection strings for the blob storage or service bus. attributes means it will be retrieved from code.
The binding key lists all bindings for this function. For this sample, it is a timer and it is the trigger, so it has a single binding. For others, it can be bindings that represents input, output and a trigger.

In the next post, I will explain the internals of each function, and how the bindings and triggers works.

How Azure Functions Blob Trigger works

Introduction

Azure functions enable you to quickly build a piece of functionality that can be triggered by an external system such as Azure Blob Storage, Storage Queue, CosmoDB, Event Hub and the list goes on. In this post, I will explain how Azure Blob Trigger works.

Sample Function

If you created a new Azure Function using Visual Studio, you will end up with the following code:

 public static class Function1
    {
        [FunctionName("Function1")]
        public static void Run([BlobTrigger("samples-workitems/{name}", Connection = "")]Stream myBlob, string name, TraceWriter log)
        {
            log.Info($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
        }
    }

The secret lies in the attribute [BlobTrigger]. It accepts the path to the storage container which Azure Web Jobs will monitor. As you know, Azure Functions is based on Azure Web Jobs, so the same triggers are used.

How Azure Web Job knows about a file that is being added, updated or removed to a container?

Run the sample app you just created in visual studio, it should work on the local storage emulator. Now, you should be able to invoke the function by creating a blob container called samples-workitems in your development storage account and you should end up with the following structure.

blob container

You can see the container “samples-workitems” which Azure Web Jobs will monitor and invoke the function whenever a file is changed there. But there is also another container named “azure-webjobs-hosts” which is the secret for how Azure monitor the files in that container.

If you opened that container, you will find a folder called “blobreceipts” which has a folder for the function name.

Inside the folder with the function name, there are some other folders with strange names like below

etags

These strange names are the ETag of each blob file added, edited or removed. So, when you change a file in any container, Web Jobs will create a new folder here with the new version ETag for that file and inside this folder you will find the same structure of the file being edited. In our case, there should be a folder called samples-workitems and inside this folder the file that was modified.

When a new file is added, Azure will check if  its ETag exist in the azure-webjobs-hosts folder and if not, then it will call the Azure Function. This way it will prevent duplicate calls for the same file. This pattern is called blob receipt.

Note that this process depends on Azure Blob Storage Logging which can take up to 10 minutes to write to the container azure-webjobs-hosts to improve performance. If you need your function to trigger faster then consider using Storage Queue trigger instead.

 

How to build a web application on Azure?

Microsoft Azure is one of the leading cloud platforms that enables us to build a scalable and highly performing web applications. In this article,  I will list the available options that allow you to quickly understand what you need in order to build your application and what services to use in each layer.

What do we need to build a web application?

Each web application needs some services in order to work propably. For example, a web server, a database server, queuing system etc..

In the past few years, I have found many people using Azure wrong or inefficiently. Some used virtual machines while they should have used Web Apps, others used a huge size of a virtual machine because they didn’t know they could scale their VM without losing the data. Accordingly, I will do my best to give you a summery of Azure services that you can utilize to build your next fantastic Web App.

Application Life-cycle Management

So, first things first. You need a place to store your use cases, test cases, source code, run unit tests and perform a continuous integration and deployment.

Although it may not be a part from Azure, but Team Services integrates seamlessly with Azure and many Azure Services like Web Apps, Mobile Apps, Azure Functions … support continuous integration with Azure.

Team Services help you to have a central repository for your source code, documents and requirement. It also supports Visual Studio Team Services and Git as a source control. It is free and supports unlimited number of projects with 5 free users.

Hosting

Now, you developed your web application, tested it and it is time to host it. Azure supports hosting for .Net, Java, PHP, NodeJS, and Python built applications.

In this phase you have more than one option:

  1. Virtual Machine: This is the very basic option and should not be used unless you have no other way, ex: migrating a legacy application that cannot work with a PAAS offering, in this case you simply have a virtual machine in the cloud and you can remotely connect to it, install the needed software and deploy your application. Note that this is the most expensive solution.
  2. Cloud Services: This is the same as Virtual Machine except that it offloads some of the work to be done by you such as setting up software and windows updates. It has 2 types, a web role which is setup to host a web application directly, and a worker role which enables you to host an executable application such as a windows service or a console app. The good part about cloud services is that you provide it with a package that has your code and it manages the deployment of the application. If the virtual machine hosting the app went down, Azure will automatically creates another virtual machine and deploy the code to it.
  3. App Service: This is the Platform as a Service or PAAS offering from Azure, all you have to worry about here is your code. It gives you a vitrual directory on cloud and all you have to do is to deploy your App to it either through web deploy, FTP or upload a package. It also offers continous integration and monitoring capabilities. This is the most flexible and cheapest option and you can autoscalre it according to the CPU usage, RAM usage and other factors. So basically, you can start small and scale as you need later. Azure App Service has Web Apps to host your application, Mobile App which is a backend as a service to your mobile apps, Logic Apps which gives you the option to build your businss logic and integration between different systems, and finally API Apps which is used to host your REST api web Apps.

Integration

If your application integrates with other systems, then you can use Azure Logic App, Azure Functions, BizTalk Service and Service Bus

Most services gives you a very nice visual designer that allows you to orchestrate your business logic and it offers out of the box seamless integration with many external systems such Office 365, Dynamic CRM or On-premise system through BizTalk Services

Data Storage

Azure offers many services to store your data, all of it are based on Azure Storage Account which is the main storage system for all services in Azure. Each account has some storage and throughput limitation, you can check these limitation from here

  1. If you need to store text files or video files, you can use Page and Block blobs. Each type is suitable for specific file types, you can read more about it from here
  2. Use Azure Media Services if you have to store and stream media files.
  3. Azure table storage is suitable if you need a NoSQL database, it provides a high throughput storing and retrieving entities that have no referential integrity between them.
  4. Azure DocumentDB is a NoSQL document database similar to MongoDB, it is a born in the cloud database that you can use if you need a low latency high performing database
  5. Azure SQL Database is a SQL Server database in the cloud where you need not to worry about SQL Server installation, Backup and Restore or anything else, you just create a database and use it.
  6. Azure File Share is used to replace any legacy File System Share in any legacy application.

 

Now, you should be able to have some basic knowledge about the features you can use to build your next web application.

Azure Services and Components

In the previous post What is Microsoft Azure? we introduced azure and cloud computing in general, we also explained how can we signup for an account using either the free trial or MS Dev Essentials.

In this post, I will explain the different parts and services in Azure.

How are services grouped in Azure?

Azure has different services for different needs, it can be organized in many ways as below

Services

  1. Compute
  2. Storage
  3. Networking
  4. Integration

Layers

  1. Application
  2. Data
  3. Integration

We can also divide it according to the consumption model (IAAS, PAAS, or SAAS), but we will stick to the above categorization for now

In this post, we will only explain the services part

Compute

This category has many wonderful services, like Virtual machines, Cloud Services which is a type of a virtual machine but more manageable by Azure and it has 2 types, web role used to host web sites and a worker role to host other code that runs on any windows machine.

Compute also has App service which as a developer you will be always using it as it offers everything you can wish for, starting from databases to DevOps, App Service has the following features

  1. Web App, similar to cloud service but it is more PAAS and lightweight, the same web app can be used to host both a web application and a timer job using Web Jobs, note that it has been always called Azure web sites, so if you heard that name, don’t be confused, it is the same
  2. Mobile App, as the name implies, it is a backend as a service for your mobile application, it offers a lot of features like a storage for your data, authentication, offline sync and push notification, it has also been known by Azure Mobile Services for long time
  3. Logic App, used to build a workflow applications to handle different business logic scenarios, ex: receive a message, post an order to the inventory service, sends an email, assign a task to shipping employee …, the logic app is based on BizTalk services which is an online version from BizTalk server
  4. API App, used to host REST APIs with a lot of features like throttling, payment for usage, reports …

Storage

This is one of the most important services in Azure and without it, you can nearly do nothing, this is used for storing anything starting from simple files, VM VHD files to databases.

Storage is divided into storage accounts, each account can hold upto 500 TB, with some limits on the number of operations per second, to know more about Azure services limits you can read more from this link

You can create more than one account and use it for different services, ex: you can have an account for your databases and another for virtual machines , but you have to take care of the limits to avoid any issues

Networking

Azure enables you to create VPN  just like on-premise and only virtual machines, web apps or any other services joined to the VPN can see each other, not only this, but you can join the VPN to on-premise VPN in any of the following ways

  1. Site to Site: this will join a whole VPN on-premise to Azure VPN, think of it as if you are expanding your datacenter
  2. Point to Site: this will join a single machine on-premise to Azure VPN

Integration

This feature allows you to integrate applications deployed on Azure with other applications running on-premise or even on Azure using BizTalk services.

 

There are a lot of other services like Azure Active Directory, SQL Database, DocumentDB, Search, I will write about these services in next post, then I will start a series where we will build an application and utilize most of azure services in one place.

 

If you have any suggestions about it, please leave a comment.

 

What is Microsoft Azure?

During the Colleague, I remember people were talking about renting the schools’ servers during the summer as there was nothing to do with it while there is no study, so why not to make some money from those people who don’t want to purchase new hardware to get their application working quickly.

Later, there were those vendors who give you a free or paid site that you can upload your files and program to it and manage it through a control panel, some times it was provided with a database engine like MySQL, sometimes not.

Nowadays, it is all about Cloud, and before we talk about Azure, lets first see what is cloud:

What is Cloud?

According to searchcloudcomputing, cloud is

Cloud computing enables companies to consume compute resources as a utility — just like electricity — rather than having to build and maintain computing infrastructures in-house.

Think of it with a real life example, people can buy a car or a home, and pay hundreds of thousands and maybe millions to purchase their dream home, they will be responsible to maintain it and fix any issues that may happen to it.

Others may just rent the home and start quickly and save their selves a great deal of money and time, still they will have to maintain the home, clean it and fix or pay for any damages.

Some other people would simply rent a room inside a hotel and enjoy their  time, have a nice room service and a good breakfast, pay for only the nights they spent and if they didn’t like it, they can always move somewhere else.

The same Idea applies for software, You can either buy your own hardware and Software to build your own data center, or you can just consume it from somewhere else, this consumption model comes in 3 forms as explained below:

Infrastructure as a Service (IAAS)

In the OnPremise model, you can notice that you manage everything, starting from the server rooms, Hardware, electricity, A/C, Storage and everything including the persons who maintain and support it.

Imaging the time needed to purchase the hardware, preparation and OS installations, batching…, all of these activities can take months to start not to mention how much it will take to finish

In real life, this is a guy who pay a lot of money to a buy a new car, he will search for a new model, think about resale price, maintenance fees, running costs, and insurance cost.

Instead, you can just contact a cloud vendor, ask them to rent a 5 or 10 virtual machines and it can be ready in few ours for you to use in your application.

In this model, you only manage the OS updates and your application + Data, you don’t have to worry about the hardware, networking or Storage.

You will not install the OS, you will just install the software you need, ex: SQL Server, IIS, Domain Controllers …

In real life, This is a guy who went to a car agency, rent a car, pay the rent and take it for a ride, he still has to put pay for Gas, wash and some other few things, but not as much as if it was his.

Platform as a Service (PAAS)

If you see that managing the virtual machines will be time consuming and hard to do, then you can go for the PAAS model, where you will be managing only your application and data.

You will need to just deploy your application, think of it as if you have a remote connection to a virtual directory on IIS where you can upload your asp.net site, and a connection to a SQL server where you can run the database scripts, you don’t have to worry about the SQL Cluster or backup, windows updates or anything else, the vendor will manage all of that for you, you just need to take care of your application.

in real life, this is a guy who rented a car with a driver, he still pay but he is doing less work now and he can focus on his own stuff instead of driving himself

Software as a Service (SAAS)

The final model is when you are not developing an application and you don’t want to do anything extra, just use the software as is with minor customization, this model is used a lot nowadays, the most used example is email service, when we need an email account, we open Hotmail, yahoo or GMail and register for a new account and start to use instantly, everyone use it as if the software was designed for him and no one can see the data for others, we never cared about where the data is stored or who is running what, we just open our browser or use a mobile application to send and receive emails.

In real life, this is a guy who simply took a taxi or Uber :). 

If you compared the 4 models (including On Premise) and the 4 real life cases, you will find the difference is mainly about:

  1. Speed: the time to start and get the environment ready is much less when you move from OnPremise -> IAAS -> PAAS -> SAAS
  2. Cost: the cost is much less, you don’t have to pay a large amount of money in advance, you simply pay for what you use, if you didn’t like the service, you can switch to another, the other reason is: for OnPremise, you will have to purchase the needed hardware for the max number of users who will use the system even if the system will be idle most of the times, so if your application is used by 10 user most of the time but in some days it is being used by 1000, you will have to purchase a hardware that support the 1000 users, but with Cloud, you can simply get resources for the 10 users and scale for more when needed.
  3. Scalability: Most cloud vendors have automatic scalabiity depending on CPU usage, Memory consumption or any other metric, so you can set your application to add an extra virtual machine if the CPU usage exceeded 80% for more than 30 minutes and remove it if it drops under 50% for 30 minutes.
  4. Manageability:  As you can see from the image, when you move to the right, the stuff you manage is less, this gives you the opportunity to put your focus on the applications you build.

What is Azure?

Microsoft Azure which is also known as Windows Azure is Microsoft Cloud Platform, it offers a great deal of features and it is growing unbelievably fast, every week or maybe every day they add more and more features and more hardware resources, when I started learning Azure, they only had 6 data centers, today they have 19 data centers, and by a data center we are talking about a place that is larger than a football field that has millions of servers read for you to use.

What is in it for me?

Azure support all the 3 types (IAAS, PAAS and SAAS), you can use it build virtual machines, you can establish a VPN between Azure and your OnPremise servers to extend your data center, you can use Azure App Service to build web sites, Mobile back end, API and Logic Apps, You can use its data services like SQL Azure, DocumentDB or Hadoop

The post will not even be enough to list all the features in Azure today, I shall write another post that introduces most of the features that you can use to build web applications with links to help you start

How to Start?

Microsoft offers a 30 days free trial with a credit of $200, after it expires you can create a subscription and still can use a lot of features offered for free.

You will notice that many services in Azure has a FREE plan with some limited features but you can use it to practice, of course it will not be possible to use these free features for production sites, one of few examples is Azure web App , it gives you 10 free Web Apps that can be accessible over the internet, it can run asp.net, NodeJS, PHP, Java and Python.

If you don’t already have an account, go to http://azure.microsoft.com/ click the Try for Free button and register.

Another way to have an account that gives you a free $25/month is through Microsoft newly introduced Dev Essentials, you just need an MS Account and then activate your benefits, if you didn’t do that already, you can do that from the below link, it also gives you a free 6 months PluralSight Subscription

Visual Studio Dev Essentials

I hope the post is helpful and if you have any comments, questions or suggestion,  please leave me a comment

In the coming posts, I will introduce Azure different services and explain how to build an application with it.