A tinkering journey diary https://andreas-wilm.github.io “Can we predict T cell specificity with digital biology and machine learning?” was the title of a recent publication in Nature Reviews Immunology by Hudson et al.. The author’s conclusion was: It’s tough. Why bother in the first place? Because T cells target all sorts of diseases, their receptor sequences... Tue, 24 Oct 2023 00:00:00 +0800 https://andreas-wilm.github.io/2023-10-24-using-ml-to-predict-tcell-specificity-using-phenotypes/ https://andreas-wilm.github.io/2023-10-24-using-ml-to-predict-tcell-specificity-using-phenotypes/ After a recent job change, I’m back at science! Yay! With this new job come analysis demands for huge next-gen sequencing data-sets. For orchestration of the analysis workflow I started out using Snakemake, simply because it’s easier to use than the alternatives, at least for me. However, I knew that... Mon, 03 May 2021 00:00:00 +0800 https://andreas-wilm.github.io/2021-05-03-nextflow-on-aws-batch/ https://andreas-wilm.github.io/2021-05-03-nextflow-on-aws-batch/ In my two previous posts I showed how to run Snakemake on Azure Kubernetes Service (AKS) without a shared filesystem. While there is nothing wrong with this setup, there is another Azure service that’s actually meant for this type of batch computing scenario: Azure Batch, a cloud based job scheduling... Thu, 30 Jul 2020 00:00:00 +0800 https://andreas-wilm.github.io/2020-07-30-azure-batch-in-snakemake/ https://andreas-wilm.github.io/2020-07-30-azure-batch-in-snakemake/ In a previous post I showed how to run Snakemake on an auto-scaling Kubernetes cluster without shared filesystem on Azure. There I used a public Dockerhub repo for the Snakemake container. Since we were running Kubernetes on Azure it makes sense to use Azure Container registries (ACR; which supports Singularity... Thu, 11 Jun 2020 00:00:00 +0800 https://andreas-wilm.github.io/2020-06-11-snakemake-on-aks-with-acr/ https://andreas-wilm.github.io/2020-06-11-snakemake-on-aks-with-acr/ Edit (Aug 2020): This has made it into the official Snakemake documentation. Refer to the official AKS executor tutorial Snakemake and Nextflow are two very popular bioinformatics workflow management systems. They allow to write analytics workflows that seamlessly scale, are reproducible, reentrant and make it easy to move between compute... Mon, 08 Jun 2020 00:00:00 +0800 https://andreas-wilm.github.io/2020-06-08-snakemake-on-ask/ https://andreas-wilm.github.io/2020-06-08-snakemake-on-ask/ This post was written jointly with Auda Eltahla, the Microsoft Research Engagement Manager for the Asia region. The content is based on this excellent blog post by Sam Cogan, which contains further notes on how to use locking to prevent subsequent changes (e.g. reactivation) of shut down resources. Overview One... Mon, 01 Jun 2020 00:00:00 +0800 https://andreas-wilm.github.io/2020-06-01-enforce-budgets/ https://andreas-wilm.github.io/2020-06-01-enforce-budgets/ Since April 2019 Azure Container Registry (ACR) supports storing of Singularity images. Today was the first time I actually had to use it and here I quickly discuss how. Once I had installed Singularity, I pulled an example image from Singularity’s cloud library. To push this image to ACR, you... Wed, 25 Mar 2020 00:00:00 +0800 https://andreas-wilm.github.io/2020-03-25-singularity-and-azure-container-registry/ https://andreas-wilm.github.io/2020-03-25-singularity-and-azure-container-registry/ For an upcoming workshop I was looking for an interesting mix of genomics and machine learning to show off Azure capabilities. My MS Genomics colleague Erdal Cosgun recently showed me his workshop material, which uses MS Genomics to predict variants and Azure Machine Learning Studio to predict (fake) phenotypes for... Wed, 04 Sep 2019 00:00:00 +0800 https://andreas-wilm.github.io/2019-09-04-azure-automl-on-genomics-data/ https://andreas-wilm.github.io/2019-09-04-azure-automl-on-genomics-data/ I recently had the immense pleasure to work on a project with Astrophysicists at ICRAR in Perth. These folks need some serious compute power to massage the ocean of data that their telescopes produce. Normally this type of analysis is a first-class application for supercomputing centers, however, supercomputing centers are... Fri, 05 Jul 2019 00:00:00 +0800 https://andreas-wilm.github.io/2019-07-05-slurm-cluster-with-lustre/ https://andreas-wilm.github.io/2019-07-05-slurm-cluster-with-lustre/ In a previous post I looked at document types in Microsoft Academic Graph (MAG), partially just to tinker with Databricks and U-SQL. Surprisingly, at least to me, a lot of documents in MAG are of unknown type. This is also plainly visible from the MAG publication website: One speculation is... Wed, 19 Jun 2019 00:00:00 +0800 https://andreas-wilm.github.io/2019-06-19-unknown-doctypes-in-mag/ https://andreas-wilm.github.io/2019-06-19-unknown-doctypes-in-mag/ Baby steps with U-SQL and Databricks - I’ve had a chance to play with Microsoft Academic and Microsoft Academic Graph (MAG) for a while and wanted to document some baby steps here . Describing the two “services” in detail is stuff for another post, suffice it to say that Microsoft Academic is is like Google Scholar, just... Fri, 31 May 2019 00:00:00 +0800 https://andreas-wilm.github.io/2019-05-31-doctypes-in-microsoft-academic-graph/ https://andreas-wilm.github.io/2019-05-31-doctypes-in-microsoft-academic-graph/ Achievement unlocked! - Since today I’m officially a Microsoft Azure Solutions Architect. All it took was two months of drinking from the fire-hose, ha! To get there I needed to pass the AZ-300 (Microsoft Azure Architect Technologies) and AZ-301 (Microsoft Azure Architect Design) exams, which cover an awful lot of ground. Retrospectively I’d... Thu, 30 May 2019 00:00:00 +0800 https://andreas-wilm.github.io/2019-05-30-azure-solutions-architect/ https://andreas-wilm.github.io/2019-05-30-azure-solutions-architect/ Azure Quickstart Templates offer a convenient way of provisioning resources on Azure using community maintained templates. Today I played with the Simple Ubuntu Linux VM template and deployed it by using both, the Azure Portal and the CLI. The simplest way to deploy a template is by hitting the “Deploy... Thu, 25 Apr 2019 00:00:00 +0800 https://andreas-wilm.github.io/2019-04-25-arm-template-default-values-in-cli/ https://andreas-wilm.github.io/2019-04-25-arm-template-default-values-in-cli/ Welcome to my tinkering journey - Well, hello! I started this blog to write about Cloud computing, Research and Frickeleien (a wonderful word from the Rhineland). Posts are mainly meant to help me digesting the cool things I learned. Some of them will hopefully be helpful to others. Happy frickeling! Wed, 24 Apr 2019 00:00:00 +0800 https://andreas-wilm.github.io/2019-04-24-aller-anfang-ist-schwer/ https://andreas-wilm.github.io/2019-04-24-aller-anfang-ist-schwer/