Month: September 2024

When developing with Azure Storage, it can significantly speed up your development process if you can use a local development environment, rather than constantly connecting directly to storage in Azure itself. This is where Azurite comes in – Azurite is a local storage emulator for Azure Storage, mimicing blob/container, queue and table storage. While there are lots of ways to get it running (e.g. from binary, manually using Docker etc.), I wanted to set it up as a service container in my devcontainer configuration which provides a few benefits:

• consistency – every time i rebuild my devcontainer i know i’m resetting Azurite back to a known, clean state
• Isolation – running it in a separate container means i avoid any potential side effects which might arise if it’s running in my main development container
• Portability – it works consistently on Github codespaces, local devcontainer setups etc.

This guide provides a basic outline for setting up Azurite as a service container in a devcontainer configuration.

Create docker-compose.yml to define the services

Using a docker compose file inside your devcontainer definition lets you define multiple ‘services’ which all work together. In my case, i’ve set up the devcontainer service which is the main development environment based on the Microsoft Python devcontainer image, and azurite which contains azurite. Of course you could add whatever you need – PostgreSQL, Mongo, whatever you need. Here’s my docker-compose file:

services:
  devcontainer:
    image: mcr.microsoft.com/devcontainers/python:1-3.11-bullseye
    platform: linux/amd64
    volumes:
      - ..:/workspace:delegated
    ports:
      - "5000:5000"
    environment:
      - POETRY_VIRTUALENVS_IN_PROJECT=true
    command: /bin/sh -c "while sleep 1000; do :; done"
    network_mode: "host"
  azurite:
    image: mcr.microsoft.com/azure-storage/azurite
    ports:
      - "127.0.0.1:10000:10000"
      - "127.0.0.1:10001:10001"
      - "127.0.0.1:10002:10002"
    command: >
      azurite
      --blobHost 0.0.0.0
      --queueHost 0.0.0.0
      --tableHost 0.0.0.0

Create devcontainer.json

After defining your services, you need to create a devcontainer.json file. This configures the development environment, and its in the dockerComposeFile attribute that you point to your docker-compose file, and defining the service which represents the actual devcontainer. I’ve added a bunch of features which . In my configuration, i’ve also disabled some extensions which are recommended by the base image or other feature extensions by prefixing their names with -. Finally, i included a postCreateCommand which marks the /workspace folder as safe, and installs dependencies using Poetry, although you may want to skip this last step.

{
	"name": "Python 3",
	"dockerComposeFile": ["docker-compose.yml"],
	"workspaceFolder": "/workspace",
	"service": "devcontainer",
	"features": {
		"ghcr.io/devcontainers-contrib/features/poetry:2": {"version": "1.8.3"},
		"ghcr.io/devcontainers/features/github-cli:1": {},
		"ghcr.io/devcontainers/features/node:1": {},
		"ghcr.io/devcontainers/features/azure-cli:1": {},
		"ghcr.io/flexwie/devcontainer-features/pulumi:1": {},
		"ghcr.io/prulloac/devcontainer-features/pre-commit:1": {},
		"ghcr.io/jlaundry/devcontainer-features/azure-functions-core-tools:1": {},
		"ghcr.io/devcontainers/features/rust:1": {}, // for cryptography package
		"ghcr.io/devcontainers/features/docker-outside-of-docker": {}
	},
	"customizations": {
		"vscode": {
			"extensions": [
				"-vadimcn.vscode-lldb",
				"-rust-lang.rust-analyzer",
				"-tamasfe.even-better-toml",
				"-dbaeumer.vscode-eslint"
			]
		}
	},
	"forwardPorts": [10000, 10001, 10002],
  "postCreateCommand": "git config --global --add safe.directory /workspace && poetry self add poetry-plugin-export && poetry config warnings.export false && poetry config virtualenvs.in-project true --local && poetry install --with dev --no-interaction --sync --no-root"
}

Next steps

You can connect to the locally emulated storage using a connection string based on a well-known account and key baked in to Azurite:

DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;

You might choose to add more advanced configuration, such as persistent storage, or enabling OAuth (to allow the use of the DefaultAzureCredential).

Finally, you can add additional services, such as PostgreSQL or Mongo, to aid your development process.

In Pulumi, when calling pulumi_azure_native.storage.list_storage_account_service_sas_output() to generate a SAS, you pass the required permissions to the permissions: Input[str | Permissions | None] parameter. pulumi_azure_native.storage.Permissions is an enum, offering simple selections (R, L etc.):

# Create a shared access signature scoped to the container
app_container_signature = (
  pulumi.Output.all(resource_group.name, storage_account.name, app_container.name)
  .apply(
    lambda args: azure_native.storage.list_storage_account_service_sas_output(
      resource_group_name=args[0],
      account_name=args[1],
      protocols=azure_native.storage.HttpProtocol.HTTPS,
      shared_access_start_time="2022-01-01",
      shared_access_expiry_time="2030-01-01",
      resource=azure_native.storage.SignedResource.C,
      permissions=azure_native.storage.Permissions.R,
      content_type="application/json",
      cache_control="max-age=5",
      content_disposition="inline",
      content_encoding="deflate",
      canonicalized_resource=f"/blob/{args[1]}/{args[2]}",
    )
  )
  .apply(lambda result: pulumi.Output.secret(result.service_sas_token))
)

But you can also pass a string of permissions, any of R, L, D, W, C, A, or P, depending on the actions you want to allow for the SAS. This allows you to specify permissions for reading (R), listing (L), deleting (D), writing (W), creating (C), adding (A), or processing (P) blobs within the container, such as permissions="RWL":

app_container_signature = (
  pulumi.Output.all(resource_group.name, storage_account.name, app_container.name)
  .apply(
    lambda args: azure_native.storage.list_storage_account_service_sas_output(
      resource_group_name=args[0],
      account_name=args[1],
      protocols=azure_native.storage.HttpProtocol.HTTPS,
      shared_access_start_time="2022-01-01",
      shared_access_expiry_time="2030-01-01",
      resource=azure_native.storage.SignedResource.C,
      permissions="RWL",
      content_type="application/json",
      cache_control="max-age=5",
      content_disposition="inline",
      content_encoding="deflate",
      canonicalized_resource=f"/blob/{args[1]}/{args[2]}",
    )
  )
  .apply(lambda result: pulumi.Output.secret(result.service_sas_token))
)

Another note for myself. I wanted to use this to give my app access to the entire account. I thought they would be a property of pulumi_azure_native.storage.StorageAccount but they’re not. Instead you need to call pulumi_azure_native.storage.list_storage_account_keys().

import pulumi
import pulumi_azure_native as azure_native
config = pulumi.Config()
# Create a Resource Group
resource_group_name = config.require("resourceGroupName")
location = config.require("location")
resource_group = azure_native.resources.ResourceGroup(
  resource_group_name, resource_group_name=resource_group_name, location=location
)
# Create a Storage Account
storage_account = azure_native.storage.StorageAccount(
  config.require("storageAccountName"),
  resource_group_name=resource_group.name,
  sku=azure_native.storage.SkuArgs(
    name=azure_native.storage.SkuName.STANDARD_LRS,
  ),
  kind=azure_native.storage.Kind.STORAGE_V2,
  location=resource_group.location,
)
# fetch primary key
storage_account_primary_key = (
  pulumi.Output.all(resource_group.name, storage_account.name)
  .apply(lambda args: azure_native.storage.list_storage_account_keys(resource_group_name=args[0], account_name=args[1]))
  .apply(lambda accountKeys: pulumi.Output.secret(accountKeys.keys[0].value))
)

In a storage account, you can create a SAS scoped to a specific container, however, that SAS does not have permission to execute BlobClient.exists() as this requires at least list privileges on the parent object (e.g. the account), and when you try to perform the check, you’ll get this error:

azure.core.exceptions.HttpResponseError: This request is not authorized to perform this operation.

Note that this is an HttpResponseError, not a ClientAuthenticationError (which is actually a more specific error which extends HttpResponseError), so you need to interrogate the specific response code, although note that if the container client does not exist, you might also get a ClientAuthenticationError with this message:

Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:a52b0bdf-401e-0030-755f-fc3a0f000000\nTime:2024-09-01T11:09:19.0123125Z\nErrorCode:AuthenticationFailed\nauthenticationerrordetail:Signature did not match. String to sign used was rwl\n2022-01-01T00:00:00.0000000Z\n2030-01-01T00:00:00.0000000Z\n/blob/<account>/<container>\n\n\nhttps\n2015-04-05\nmax-age=5\ninline\ndeflate\n\napplication/json

As a workaround, you can instead try to read from or write to (depending on the privilege level you want to test for) the container. For example, you might do something like this:

def test_if_container_exists(container_client: ContainerClient) -> bool:
  """check if a container exists first by calling the .exists() method, even if the SAS scope is restricted to the specific container"""
  # first use the .exists() method to check if the container exists
  try:
    container_client.exists()
    return True
  except ClientAuthenticationError:
    # we dont know if this is because the SAS token doesnt have access or the container doesnt exist
    pass
  except HttpResponseError as e:
    if e.status_code == 404:
      return False
    elif e.status_code == 403:
      pass  # could be a SAS token scope restriction
  except ResourceNotFoundError:
    return False
  except Exception as e:
    # if we got any other exception, raise it
    raise
  # if we got ClientAuthenticationError, try to write a small blob to check if the SAS token is valid
  try:
    blob_client = container_client.get_blob_client("test_blob")
    blob_client.upload_blob(b"Test content", overwrite=True)
    return True
  except ClientAuthenticationError or ResourceNotFoundError:
    return False
  except Exception as e:
    # if we got any other exception, raise it
    raise