The Docker CLI is not very user-friendly, you have to be explicit about lots of options that actually make sense as defaults in everyday usage and debugging when we are working in a multi-container environment. …

If you are an entrepreneur, you will be successful if you build what solves customers’ problems. You need to be in their shoes, understand their short-term and long-term requirements, need to understand how is their day-to-day life, what problems do they face; and then based on it, prepare short-term and…

Terraform is IaC (Infrastructure as Code) tool to provision and manage any cloud, infrastructure, or service. It follows a declarative-style and push-type model. It is masterless by default and does not require you to install any extra agents. Disclaimer: This is going to be a lengthy writeup, as we are…

Writing and debugging Jsonnet code is not straight-forward, but let’s see what we have at our disposal. As a data scientist and a software engineer, I have been using YAML files for defining jobs to train machine learning models. Recently, I switched to, data templating lazy language by Google, to…

Instead of TDD (Test Driven Development) which forces you to think about tests first, one can practice TPD (Test Paralleled Development). …

Orchestrate parallel jobs on K8s with the container-native workflow engine. Table of Contents ∘ Argo CLI ∘ Deploying Applications ∘ Argo Workflow Specs Argo Workflows is an open-source container-native workflow engine for orchestrating parallel jobs on K8s. Argo Workflows are implemented as a K8s CRD (Custom Resource Definition). As a result, Argo workflow…

JSON Schema and OpenAPI can seem similar but have different use-cases. To begin, how JSON Scheme and OPenAPI differ? Well, in contrast to JSON Schema, an OpenAPI document is a definition for an entire API, not just data models. One might compare JSON Schema with the OpenAPI data model. Why the need to validate JSON? There…

Don’t get confused with these two similar but different patterns, and know which one to use when. This difference is important to not just generic Software Engineers, but also to Data Engineers and is the basis for the understanding event-driven architectures for data pipelines. Let’s look at both of them…