By default, Kafka attempts to send records as soon as possible, sending up to max.in.flight.requests.per.connection messages per connection. If you attempt to send more than this, the producer will start batching messages, but ultimately if you saturate the connection so there are unacknowledged message batches pending, the producer enters blocking mode. We can optimise the way the producer batches messages to achieve higher throughput through two settings, batch.size and linger.ms. linger.ms is the number of milliseconds that the producer waits for more messages before sending a batch. By default, this value is 0, meaning that the producer attempts to send… Continue reading →
Claude Code is pretty amazing. It’s let me build prototypes and improve apps faster than I ever thought possible. But I was wondering – what else can it do? I’ve been using Emby for years, but recently I started to wonder what happened to Jellyfin, the project that forked Emby years ago. So i decided to ask Claude Code to help me by setting up a debate and mediating the answer. Setting up the debate Here’s the prompt i used: pretty basic stuff. It got to work, setting up a task list: The folder structure was quite basic: debate.md has… Continue reading →
Although builds were succeeding locally, during CI builds were failing on Linux with the error Error: Cannot find module @rollup/rollup-linux-x64-gnu. npm has a bug related to optional dependencies. The log suggested removing package-lock.json – but this obviously breaks deterministic builds. The root cause is a missing dependency – specifically, when Rollup needs native binaries for performance optimisation. Mac-generated lockfiles won’t include Linux-specific optional dependencies. First noted in npm bug 4828, it was fixed in npm 11.3.0+, but adding an explicit optional dependency prevents edge cases. The fix is: This allows builds to work reliably across platforms. Continue reading →
I was recently asked to design a method to meet ITAC (IT Application Controls) standards for critical data flows in our organisation. ITAC are application-level controls looking mainly at how we ensure the completeness, accuracy and validity of transactions – for example, invoices or trades. Our control set is based on the ICFR principles, of which ITAC is one part. The specific control objective I was asked to look at relates to the risk of loss of integrity of financial data transfers. The focus on the integrity of the data, not authenticity or non-repudiation is really important as it means… Continue reading →
In a previous post, I explained how to configure Azurite to use a self-signed certificate to enable OAuth authentication. One challenge with this method is that the Azure Python SDK will refuse to connect to azurite, reporting errors such as: This is because the the Azure SDK uses the Requests library, which in turn uses Certifi as its source of root certificates. Certifi provides a regularly updated bundle of the Mozilla root trust store, and our self-signed custom certificate obviously isn’t in Mozilla’s trust list! Making Requests trust our self-signed certificate You can get around this by setting the REQUESTS_CA_BUNDLE… Continue reading →
I’ve been using Azurite to simulate Azure storage for my development. If you’re not familiar with it, Azurite is a local storage emulator for Azure Storage, and you can read my other post about how i’ve set up my devcontainer configuration to run Azurite as a service container. As my deployed code is using an Azure Managed Identity, I wanted ensure my development environment was consistent with this and also uses Azure DefaultAzureCredential credential provider class. In this post, i will talk through the steps required to switch from using a connection string (with a well-known account and key) to… Continue reading →
When developing with Azure Storage, it can significantly speed up your development process if you can use a local development environment, rather than constantly connecting directly to storage in Azure itself. This is where Azurite comes in – Azurite is a local storage emulator for Azure Storage, mimicing blob/container, queue and table storage. While there are lots of ways to get it running (e.g. from binary, manually using Docker etc.), I wanted to set it up as a service container in my devcontainer configuration which provides a few benefits: This guide provides a basic outline for setting up Azurite as… Continue reading →
In Pulumi, when calling pulumi_azure_native.storage.list_storage_account_service_sas_output() to generate a SAS, you pass the required permissions to the permissions: Input[str | Permissions | None] parameter. pulumi_azure_native.storage.Permissions is an enum, offering simple selections (R, L etc.): But you can also pass a string of permissions, any of R, L, D, W, C, A, or P, depending on the actions you want to allow for the SAS. This allows you to specify permissions for reading (R), listing (L), deleting (D), writing (W), creating (C), adding (A), or processing (P) blobs within the container, such as permissions=”RWL”: Continue reading →
Another note for myself. I wanted to use this to give my app access to the entire account. I thought they would be a property of pulumi_azure_native.storage.StorageAccount but they’re not. Instead you need to call pulumi_azure_native.storage.list_storage_account_keys(). Continue reading →
In a storage account, you can create a SAS scoped to a specific container, however, that SAS does not have permission to execute BlobClient.exists() as this requires at least list privileges on the parent object (e.g. the account), and when you try to perform the check, you’ll get this error: Note that this is an HttpResponseError, not a ClientAuthenticationError (which is actually a more specific error which extends HttpResponseError), so you need to interrogate the specific response code, although note that if the container client does not exist, you might also get a ClientAuthenticationError with this message: As a workaround,… Continue reading →
I’m rob. I spend my time exploring the world, playing board games with my family, solving complex technical problems, and learning new things. At work, I lead a team of solution architects designing and building complex realtime trading systems. Sometimes i write about things here, or code them on GitHub. I believe a few things that guide what I do and how I do it: