Configuring autoscaling for Intelligent Voice on Microsoft Azure.  The steps apply for both Ubuntu and Redhat.

Pre-requisites

The App Node VM should have the System Assigned Identity turned ON:

Step 1. Create an Application Insight resource

This is required to be able to see the custom metrics when creating the autoscaling rules.

1. Sign in to the Azure portal at https://portal.azure.com

2. Click on Create a resource in the upper left-hand corner of the Azure portal then click on Developer Tools and choose Application Insights
   
   

3. Enter a Name and Node.js Application as an Application Type. Fill the rest of the fields then click Create
   
   

   

4. Open the new Application Insight Resource then copy the Instrumentation Key
   

Step 2. Install GearmanStats on the App Node VM

GearmanStats service should be installed on the App Node VM to collect the custom metrics from Gearman and send them to Azure Monitor.

Install GearmanStats for IV 6.1+

For IV 6.1 or above, you'll find gearman_stats in the installer. Please add the followings to installer config (hosts.ini) and re-run the installer.

gearman_stats_enabled=true
gearman_stats_platform=azure
instrumentation_key=<instrumentation_key copied from application insights>

Install GearmanStats for IV 6.0 or before

1. Configure the fields as below in common-config in the gearman-stats-standalone installer:

   iv_docker_registry=<IV container registry>
   iv_docker_registry_user=<credential to IV container registry>
   iv_docker_registry_passwd=<credential to IV container registry>
   
   host_name=<private ip address of app node>
   
   install_cloudwatch_logger="N"
   install_azure_monitor_logger="Y"
   
   instrumentation_key=<instrumentation_key copied from application insights>

2. Run the installation script

   cd installation_scripts/
   chmod +x ../**/*.sh
   ./install.sh

Step 3. Create images of the Proc Node VMs

This step is to create VMs for the separate Proc Node workers, following the installation guide, which will then be turned into images which you can launch multiple copies of.

The recommended approach, particularly for larger systems, is to create individual images for each worker type (ASR, VAD, Tagger, Cracker, etc)

For example, create a VM that only has the ASR Worker running on it and validate that it connects correctly the App Node VM and can process data.

Once it has been setup correctly follow these steps to create its image:

1. Deprovision the VM

   1. Connect to your Linux VM with an SSH client

   2. In the SSH window, enter the following command:
      

      sudo waagent -deprovision

   3. Enter y to continue
      

   4. After the command completes, enter exit to close the SSH client

2. Create VM Image either in the portal or with Azure CLI etc - for example by following this guide: https://docs.microsoft.com/en-us/azure/virtual-machines/capture-image-portal

Step 4. Create a virtual machine scale set (for each Proc Node)

1. Sign in to the Azure portal at https://portal.azure.com

2. Click Create a resource in the upper left-hand corner of the Azure portal. Search for scale set, choose Virtual machine scale set, then select Create 
   
   
   

3. Enter a name for the scale set

4. Select Azure Spot instance with Eviction Type = Capacity Only and Eviction Policy = Delete

5. Select Instance Size according to the needs (eg. ASR = Standard_NC4as_T4 for GPU or VAD = Standard_B4ms)

6. For the Operating system disk image click on Browse all images then click on My Items tab and select the Image created in the above section Create images of the Proc Node VMs
   

7. Enter the Resource Group and Location

8. Enter a Username and Password or SSH public key

9. Enter the Instance count and select the Instance size needed

10. Choose None for Load balancing options

11. Select the Virtual Network (carefully select the current active network, otherwise need to create scale set again)

12. Enable Overprovisioning in the Management tab

13. Click Create

Step 5. Set the Autoscale Rules (for each VM Scale Set)

1. Sign in to the Azure portal at https://portal.azure.com

2. Click the Azure Monitor icon in the left pane

3. Click Autoscale to view all the resources for which Autoscale is applicable, along with their current Autoscale status

4. Click on the VM Scale Set created in the previous section
   

   

5. Select Custom autoscale
   

6. Enter 0 for Minimum Instance limits and Default

7. Enter the Maximum Instance limits allowed
   

   

8. Click on Add a rule on the Default Auto created scale condition to create a Scale Out rule
   

   

    

   1. Under Scale rule select Application Insights for Metric source
      
      

   2. Select the resource created in the previous section “Create an Application Insights resource” under Resource

   3. In Criteria select Last under Time aggregation

   4. Select the Metric name needed. In our example “gearmand_functions_asr_queue”

   5. Enter 1 under Time grain

   6. Select Maximum under Time grain Statistic

   7.  Select Greater than or equal to for Operator

   8. Enter 1 for Threshold

   9. Enter 5 for Duration

   10. In Action select Increase Count to under Operation

   11. Enter 1 in Instance count

   12. Enter 5 for Cool down

   13. Click Add

9. Click on Add a rule on the Default Auto created scale condition to create a Scale In rule
   

   

   1. Under Scale rule select Application Insights for Metric source

   2. Select the resource created in the previous section “Create an Application Insights resource” under Resource

   3. In Criteria select Last under Time aggregation

   4. Select the Metric name needed. In our example “gearmand_functions_asr_queue”

   5. Enter 1 under Time grain

   6. Select Maximum under Time grain Statistic

   7.  Select Less than for Operator

   8. Enter 1 for Threshold

   9. Enter 5 for Duration

   10. In Action select Decrease Count by under Operation

   11. For Instance count enter the value of the Maximum Instance limits allowed for the condition in step 8

   12. Enter 5 for Cool down

   13. Click Add

10. Add more Scale Out rules as needed. For example to increase the instance count to 2 when the asr queue in greater than or equal to 11

11. Click Save
    
    

Step 6. Turn on system-assigned Identity (on App Node only)

1. Sign in to the Azure portal at https://portal.azure.com

2. Click on Virtual Machines and select the App node VM

3. Click on Identity and Click On in Status

Step 7. Validate the setup by processing test files

Follow the validation section of the installation guide