[{"cover_image_url":"https://storage.googleapis.com/abatra-portfolio-content/blogs/bigquery_authorized_views-v2.md","updated_at":"2026-09-03T11:55:12.257540Z","category":"Data Engineering","content":"
\n\n
\n\n*By Data Potion*\n\n
\nWhen working with Google Cloud BigQuery, balancing **data sharing** with **data security** can be a challenge. We all have security measures in place, but securing data shouldn't mean blocking users from deriving valuable insights.
\n
\nBigQuery provides a wide array of options for securing your data while keeping it accessible, such as network restrictions, row-level and column-level security, and authorized datasets. In this post, we will focus exclusively on **Authorized Views** and how they facilitate secure data sharing via SQL.
\n
\n\n## The Problem: Sharing Insights Without Exposing Source Data
\n
\nImagine a scenario where multiple users want to analyze data to gain insights. However, the source data contains sensitive information—Personally Identifiable Information (PII), financial records, or unmasked confidential data.
\n
\nIf these users get direct access to the source tables, they might inadvertently gain access to information they aren't supposed to see. As data engineers, our goal is to ensure people can query the data they need without ever exposing the raw source tables.
\n
\n\n## The Solution: Authorized Views
\n
\nThe most robust way to handle this in BigQuery is by creating a **View**.
\n
\n\nA view allows you to explicitly exclude sensitive columns (like PII or financial data) from the query definition. You can then distribute these views to the users.
\n
\n\nBut how does this work under the hood?
\n* An **Authorized View** is essentially a virtual table.
\n* The view itself—not the user—is granted permission to access the source table.
\n* The user is granted the `BigQuery Data Viewer` role on the authorized dataset.
\n* The user queries the view, and the view fetches the records from the source table on their behalf.
\n\n
\nThis ensures that the end-user never interacts with the raw, potentially sensitive data.
\n
\n\n## The Vending Machine Analogy
\n\n
\nIf this sounds complex, think of an Authorized View like a **Vending Machine**:
\n
\n\n* **The Snack (Your Data):** The specific item the user wants.
\n* **The Glass & Metal Box (BigQuery Security):** The machine is locked. Nobody can just reach in and grab whatever they want.
\n* **The Inner Items (Source Tables):** The soft drinks, protein bars, and ice cream inside the machine. They are visible, but securely locked away.
\n\n* **The Keypad (Authorized View):** You input a specific request (your pin or code). The interface takes that input, processes it, and dispenses *exactly* what you asked for—nothing more.
\n
\n\n## Scaling with Authorized Datasets
\n
\nCreating a single authorized view works great for small tasks. But what happens when you have a huge number of views to share across different workloads? Granting users access to each individual view would quickly become a cumbersome, complex nightmare.
\n
\n\n**The Answer: Authorized Datasets.**
\n\n
\nInstead of managing permissions on a per-view basis, you can club all relevant views into a single dataset. You then authorize the *dataset* itself. Once a user has access to the authorized dataset, they can seamlessly query any view inside it. In turn, those views are trusted to query the raw source tables. This makes managing permissions incredibly scalable and clean.
\n
\n\n## What's Next?
\n
\nAuthorized views and datasets are powerful tools for sharing data securely in any industry—whether it's healthcare, telecommunications, or retail. Stay tuned for a future post where we will dive into a hands-on demo to see these concepts in action with an industry-specific scenario!\n
\n\n**YouTube Video**\n
\n[](https://www.youtube.com/watch?v=IpMerHSwxHM)\n","tags":["BigQuery","GCP","Google"],"title":"BigQuery Data Sharing: Mastering Authorized Views and Datasets","read_time":8,"published":true,"created_at":"2026-09-03T11:55:12.257540Z","summary":"Control Data access using Auth Views","id":"BigQuery_Data_Sharing:_Mastering_Authorized_Views_and_Datasets"},{"cover_image_url":"https://storage.googleapis.com/abatra-portfolio-assets-public/blog-img/distributed_data.png","category":"Data Engineering","updated_at":"2026-08-16T08:47:28.434123Z","content":"\nAs modern applications scale to handle increasingly higher loads, the traditional approach of simply upgrading a single machine—known as vertical scaling—quickly hits a wall of soaring costs and single points of failure[cite: 1].
\nTo build truly resilient, high-performing, and globally accessible systems, we must look toward distributed architectures and the power of shared-nothing horizontal scaling[cite: 1].
\nIn this post, we will explore the core concepts of scaling, the mechanics of keeping data replicated across multiple nodes, and the critical trade-offs involved in keeping our databases highly available and consistent[cite: 1].
\n\n
\nScaling to higher loads is a critical challenge in system design. Historically, the go-to solution was **Vertical Scaling**, also known as Scaling Up
\n\n
\n\n## Scaling Approaches \n\n
\n\n### Vertical Scaling (Shared Memory)
\n\nScaling up involves adding more RAM, CPUs, and disks to a single operating system.
\n* **Pros:** You can swap components anytime without a shutdown.
\n* **Cons:** Costs grow twice as fast, and it offers limited fault tolerance since it relies on a single geographic location.
\n\n
\n\n### Vertical Scaling (Shared Disk)
\n\nThis approach stores data in an array of disks while independent machines (with their own CPUs and RAM) connect via a fast network like NAS.
\n* **Pros:** Simplified data management, instant compute failover, independent elastic scaling, and strong ACID consistency.
\n* **Cons:** It requires complex locking mechanisms and presents a single point of failure for storage.
\n\n
\n\n### Horizontal Scaling (Shared Nothing)
\n\nHere, each node (machine or virtual machine) operates independently with its own CPU, RAM, and disk.
\n* Coordination happens at the software level over a conventional network, requiring no special hardware.
\n* It allows data distribution across multiple geographic regions, reducing latency for users and surviving entire datacenter losses.
\n\n
\n\n## Understanding Replication
\n
\n\nReplication involves keeping a copy of the same data on several different nodes connected via a network.
\nIt is useful when the dataset is small enough to fit on a single machine; otherwise, data partitioning is required.
\n\n
\n\n**Why Replicate?**
\n\n* To keep data geographically close to your users (reducing latency).
\n* To allow the system to continue working even if some of its parts have failed (increasing availability).
\n* To scale out the number of machines that can serve read queries (increasing read throughput).
\n\n
\n\nThe main difficulty in replication lies in handling changes to replicated data.
\n\n
\n\n### Leader and Follower Algorithm
\n\n
\nAlmost all distributed databases use single-leader, multi-leader, or leaderless algorithms for achieving replication.
\nIn a leader-based setup:
\n\n* One replica is designated the **leader** (primary or master). When clients want to write to the database, they must send requests to the leader, which writes new data to its local storage.
\n* Other replicas are **followers** (read replicas or slaves).
\n* The leader sending new data to local storage also sends the data change to all followers via a replication log or change stream.
\n* Followers update their local copy by applying all writes in the exact order processed by the leader database. Client read operations can be handled by either the leader or any of the followers.
\n\n
\n\n### Synchronous vs. Asynchronous Replication
\n\n
\n\n* **Synchronous:** The leader waits for the follower to confirm the write before reporting success to the user.
\n* **Asynchronous:** The leader doesn't wait for a response from the follower, and there is no guarantee of how long it might take.
\n* **Trade-offs:** Synchronous guarantees an up-to-date copy on the follower and data availability if the leader fails, but the disadvantage is that it blocks writes if a follower doesn't respond. \n\nAsynchronous doesn't block writes, but you cannot be sure data is available if the leader fails. In practice, databases usually enable synchronous replication on one follower, and keep the others asynchronous.
\n\n
\n\n## Operating a Replicated System
\n
\n\n### Setting Up New Followers
\n\n
\nSimply copying data files from one node to another isn't sufficient because clients are constantly writing to the database.
\n1. Take a consistent snapshot of the leader’s database (if possible, without taking a lock on the entire database).
\n2. Copy the snapshot to the new follower node.
\n3. The follower connects to the leader and requests all data changes since the snapshot was taken (using log sequence numbers in PostgreSQL or binlog coordinates in MySQL).
\n4. Once it has processed the backlog, it can now continue to process data changes from the leader as they happen.
\n\n
\n\n### Handling Node Outages
\n
\n\nThe goal is to achieve zero downtime.
\n* **Follower Failure (Catchup Recovery):** Easy and simple. The follower has logs for data changes on its local disk. In case of failure, it can recover from the last transaction recorded and ask the leader for data changes occurred after the last transaction.
\n* **Leader Failure (Failover):** Tricky and complex to achieve. Operations teams usually go for manual failovers over automatic ones. Automatic failover requires determining that the leader has failed, choosing a new leader, and reconfiguring the system. A lot of things may go wrong: in case of asynchronous replication, the new leader may not have received all writes; or a \"split-brain\" scenario could occur if both nodes believe they are the leader, resulting in conflicting writes. ","tags":["distributed_data"," SQL"," postgres"],"title":"Scaling to Higher Loads: The Essentials of Distributed Data and Replication","read_time":8,"created_at":"2026-08-16T08:47:28.434123Z","published":true,"summary":"Scaling is the only opton when your client wants to cater to fater requests processing with minimal lag","id":"Scaling_to_Higher_Loads:_The_Essentials_of_Distributed_Data_and_Replication<"},{"cover_image_url":"https://storage.googleapis.com/abatra-portfolio-assets-public/blog-img/OLTP_Systems.png","updated_at":"2026-07-24T12:09:16.359737Z","category":"Data Engineering","content":"### ***History | What | Why***
\n\nIn early days, a write to the database typically corresponds to a commercial transaction taking place.Making a sale, placing an order with a supplier, paying an employee’s salary, etc.\nAs databases expanded into areas that didn’t involve money changing hands, the term *transaction* nevertheless stuck, referring to a group of reads and writes that form a logical unit.
\n\nTransaction processing just means allowing clients to make low-latency reads and writes—as opposed to *batch processing jobs*, which only run periodically (for example, once per day).\nEven though databases started being used for many kinds of data—comments on blog posts, actions in a game, contacts in an address book, etc.—the basic access pattern remained similar to processing business transactions.
\n\nAn application typically looks up a small number of records by some key, using an index. Records are inserted or updated based on the user’s input. Because these applications are interactive, the access pattern became known as *online transaction processing (OLTP)*.
\n\nThese databases also started being increasingly used for data analytics, which has very different access patterns.
\nFor example, if your data is a table of sales transactions, then analytic queries might be:
\n* What was the total revenue of each of our stores in January?
\n* How many more devices than usual did we sell during our latest promotion?
\n* Which brand of baby food is most often purchased together with brand X diapers?
\n\nThese queries are often written by business analysts, and feed into reports that help the management of a company make better decisions (*business intelligence*).\nIn order to differentiate this pattern of using databases from transaction processing, it has been called *online analytic processing (OLAP)*.
\n\n
\n\n### ***OLTPs Row Based Storage Pattern***
\nTo Query this efficiently in OLTP systems indexes on `fact_sales.date_key` and/or `fact_sales.product_sk` that tell the storage engine where to find all the sales for a particular date or for a particular product.
\nBut then, a row-oriented storage engine still needs to load all of those rows (each consisting of over 100 attributes) from disk into memory, parse them, and filter out those that don’t meet the required conditions. That can take a long time.
\n\n```sql\nSELECT\n dim_date.weekday, dim_product.category,\n SUM(fact_sales.quantity) AS quantity_sold\nFROM fact_sales\nJOIN dim_date ON fact_sales.date_key = dim_date.date_key\nJOIN dim_product ON fact_sales.product_sk = dim_product.product_sk\nWHERE\n dim_date.year = 2013 AND\n dim_product.category IN ('Fresh fruit', 'Candy')\nGROUP BY\n dim_date.weekday, dim_product.category;\n```\n\n### ***Why ACID***
\n\nIn the harsh reality of data systems, many things can go wrong:
\n\n* The database software or hardware may fail at any time (including in the middle of a write operation).
\n* The application may crash at any time (including halfway through a series of operations).
\n* Interruptions in the network can unexpectedly cut off the application from the database, or one database node from another.
\n* Several clients may write to the database at the same time, overwriting each other’s changes.
\n* A client may read data that doesn’t make sense because it has only partially been updated.
\n* Race conditions between clients can cause surprising bugs.
\n\nTo be reliable, a system must deal with these faults and ensure that they don’t cause catastrophic failure of the entire system. However, implementing fault tolerance mechanisms is a lot of work.It requires a lot of careful thinking about all the things that can go wrong, and a lot of testing to ensure that the solution actually works.
\n\n### ***Transactions***
\n\nConceptually, all the reads and writes in a transaction are executed as one operation: either the entire transaction succeeds (*commit*) or it fails (*abort, rollback*). If it fails, the application can safely retry. With transactions, error handling becomes much simpler for an application.
\nBecause it doesn’t need to worry about partial failure—i.e., the case where some operations succeed and some fail (for whatever reason).
\n\n### ***Control and Locking Mechanisms***
\n\nConcurrency control is the mechanism used to ensure that multiple users or applications can interact with the database simultaneously without corrupting the data. Without concurrency control, simultaneous transactions can cause severe data anomalies, such as:
\n* *Dirty Reads:* Reading uncommitted data from another transaction that later rolls back.
\n* *Non-Repeatable Reads:* Reading the same row twice in a transaction and getting different values because another transaction updated it in between.
\n* *Phantom Reads:* Executing a query twice and seeing new \"phantom\" rows appear the second time because another transaction inserted them.
\n\nTo prevent this, relational databases primarily use *Locking Mechanisms* to serialize access to data.
\n\n### ***Lock Granularity***
\n\nDatabases don't just lock rows; they can lock data at various structural levels depending on the scope of the query.
\n* *Row-Level Locking:* Locks a single record. Excellent for concurrency (many users can edit different rows in the same table) but requires higher overhead for the database engine to manage thousands of tiny locks.
\n* *Page-Level Locking:* Locks a block of memory (usually 4KB or 8KB) that contains multiple rows. It's a middle ground between row and table locks.
\n* *Table-Level Locking:* Locks the entire table. Very low overhead, but terrible for concurrency—if a bulk update is running, no one else can read or write to that table.
\n* *Database-Level Locking:* Locks the entire database, typically only used during major structural upgrades or restores.
\n\n### ***Lock Type***
\n\n| Lock Type | Also Known As | Behaviour |\n| :--- | :--- | :--- |\n| *Shared Lock (S)* | Read Lock | Allows a transaction to read a resource. Multiple transactions can hold S-locks on the same data simultaneously. |\n| *Exclusive Lock (X)* | Write Lock | Allows a transaction to update or delete a resource. Only one transaction can hold an X-lock on data at a time. |\n
\n\n### ***ACID - Atomicity***
\n\n*Atomic* - In general, atomic refers to something that cannot be broken down into smaller parts.
\nThe system can only be in the state it was before the operation or after the operation, not something in between.
\nNote - atomicity is not about concurrency. It does not describe what happens if several processes try to access the same data at the same time, because that is covered under the letter I, for isolation.
\n\nAtomicity describes what happens if a client wants to make several writes, but a fault occurs after some of the writes have been processed.
\nSo, if the writes are grouped together into an atomic transaction, and the transaction cannot be completed (committed) due to a fault, then the transaction is aborted and the database must discard or undo any writes it has made so far in that transaction.
\n\nWithout atomicity, if an error occurs partway through making multiple changes, it’s difficult to know which changes have taken effect and which haven’t.
\nAtomicity simplifies this problem: if a transaction was aborted, the application can be sure that it didn’t change anything, so it can safely be retried.
\n\n### ***ACID - Consistency***
\n\nThe idea of ACID consistency is that you have certain statements about your data (invariants) that must always be true.
\nFor e.g., in an accounting system, credits and debits across all accounts must always be balanced.
\nHowever, this idea of consistency depends on the application’s notion of invariants, and it’s the application’s responsibility to define its transactions correctly so that they preserve consistency.
\n\nThis is not something that the database can guarantee: if you write bad data that violates your invariants, the database can’t stop you.
\nIn general, the application defines what data is valid or invalid—the database only stores it.
\nNOTE - The application may rely on the database’s atomicity and isolation properties to achieve consistency.
\n\n### ***ACID - Isolation***
\nIsolation in the sense of ACID means that concurrently executing transactions are isolated from each other: they cannot step on each other’s toes.
\nThe classic database textbooks formalize isolation as *serializability*, which means that each transaction can pretend that it is the only transaction running on the entire database.
\nThe database ensures that when the transactions have committed, the result is the same as if they had run serially (one after another), even though they may have run concurrently.
\nTypes of Isolation levels – *Read Uncommitted, Read Committed, Repeatable Reads, Serializable & Snapshot*
\n\n### ***ACID - Durability***
\nDurability is the promise that once a transaction has committed successfully, any data it has written will not be forgotten, even if there is a hardware fault or the database crashes.
\nIn a replicated database, durability may mean that the data has been successfully copied to some number of nodes.
\nIn order to provide a durability guarantee, a database must wait until these writes or replications are complete before reporting a transaction as successfully committed.
\nPerfect durability does not exist: if all your hard disks and all your backups are destroyed at the same time, there’s obviously nothing your database can do to save you.
\nIn practice, there is no one technique that can provide absolute guarantees. There are only various risk-reduction techniques, including writing to disk, replicating to remote machines, and backups—and they can and should be used together.
\n\n### ***Normalization | What | Why | Types***
\nDatabase normalization is a systematic framework for organizing data in a relational database. Its primary goal is to reduce data redundancy and improve data integrity.
\nWhile modern data warehouses and analytics engines—like BigQuery/RedShift—often favour heavily denormalized schemas (like flat tables or nested arrays) to speed up read-heavy analytical queries, the upstream transactional systems (OLTP) feeding those pipelines rely on strict normalization to maintain accuracy and prevent data corruption.
\n*WHY ? To Prevent Anomalies* – If a database is not normalized, redundant data causes structural problems known as anomalies whenever data is modified.
\nFor e.g., Update Anomaly, Insert Anomaly, Deletion Anomaly.
\n\n**Example Unnormalized Data:**
\n| EmpID | EmpName | DeptName | DeptHead | ProjectID | ProjectRole |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| 101 | Rahul | Data | Anita | P1 | Engineer |\n| 101 | Rahul | Data | Anita | P2 | Lead |\n| 102 | Sneha | Data | Anita | P1 | Engineer |\n
\n\n* *Update Anomaly* - If Anita leaves and is replaced as the Data department head, you must update multiple rows. If one row is missed, the database becomes inconsistent (Rahul and Sneha would appear to have different department heads).
\n* *Insertion Anomaly*: You cannot add a new department (e.g., \"Security\" led by \"Vikram\") until you hire an employee to put in that department, because EmpID is required for a new row.
\n* *Deletion Anomaly*: If Rahul and Sneha both finish their projects and are temporarily unassigned, deleting their rows entirely wipes out the fact that the \"Data\" department exists and is led by \"Anita\".
\n\n## ***Normal Forms***
\n* Normalization solves these anomalies by decomposing large tables into smaller, tightly focused tables linked by relationships.
\n* Normal forms are progressive rules. To achieve a higher normal form, a database must first satisfy the rules of the lower ones.
\nTypes: *1NF, 2NF, 3NF*
\n\n## ***1 NF – Unique Record & Atomic Values***
\n*The Rule:* Every column must contain atomic (indivisible) values, and each record must be unique. No repeating groups or arrays in a single column.
\n*1NF Solution:* Split the multi-valued attribute into separate rows (or a separate table, depending on the broader schema design).
\n\n**Before 1NF:**\n| EmpID | Name | Phone_Numbers |\n| :--- | :--- | :--- |\n| 101 | Rahul | 555-0101, 555-0102 |\n
\n\n**After 1NF:**\n| EmpID | Name | Phone_Number |\n| :--- | :--- | :--- |\n| 101 | Rahul | 555-0101 |\n| 101 | Rahul | 555-0102 |\n
\n\n## ***2 NF – No Partial Dependencies***
\n*The Rule:* Must be in 1NF, and all non-key attributes must be fully functionally dependent on the entire primary key. (This only applies when a table has a composite primary key).
\nConsider a table with a composite primary key of (EmpID, ProjectID):
\n* Hours depends on both the Employee and the Project.
\n* EmpName only depends on EmpID (a partial dependency).
\n* ProjectName only depends on ProjectID (a partial dependency).
\n\n*2NF Solution:* Break the partial dependencies into their own tables.
\n\n
\n\n**Table 1: Employee**\n| EmpID (PK) | EmpName |\n| :--- | :--- |\n| 101 | Rahul |\n\n**Table 2: Project**\n| ProjectID (PK) | ProjectName |\n| :--- | :--- |\n| P1 | Migration |\n| P2 | API Setup |\n\n**Table 3: Employee_Project**\n| EmpID (PK) | ProjectID (PK) | Hours |\n| :--- | :--- | :--- |\n| 101 | P1 | 40 |\n| 101 | P2 | 20 |\n\n
\n\n## ***3 NF – No Transitive Dependencies***
\n*The Rule:* Must be in 2NF, and there must be no transitive dependencies. This means non-key attributes must depend only on the primary key, and not on other non-key attributes.
\nHere, DeptLocation depends on DeptID, and DeptID depends on EmpID. Because DeptLocation relies on the primary key through another column, it's a transitive dependency. This causes the update anomaly mentioned earlier.
\n\n*3NF Solution:* Move the transitively dependent data into its own lookup table.
\n\n**Table 1: Employee**\n| EmpID (PK) | EmpName | DeptID (FK) |\n\n| 101 | Rahul | D1 |\n| 102 | Sneha | D1 |\n\n**Table 2: Department**\n| DeptID (PK) | DeptLocation |\n| :--- | :--- |\n| D1 | Pune |\n
\n\n## think. believe. grow","tags":["strong-consistency"," durable"," transactions"],"title":"Understanding OLTP Systems - Backbone of Transactional World & Robust Consistency","read_time":8,"published":true,"created_at":"2026-07-24T12:09:16.359737Z","summary":"Online Transaction Processing (OLTP) systems - designed for fast, interactive read and write database operations","id":"Understanding_OLTP_Systems_-_Backbone_of_Transactional_World_&_Robust_Consistency"},{"cover_image_url":"https://storage.googleapis.com/abatra-portfolio-assets-public/blog-img/3_pillars.png","category":"data engineering","updated_at":"2026-06-19T12:34:52.717000Z","content":"As engineering leaders and architects, we are constantly challenged to build **unstoppable**, high-performance systems. Today's data-driven landscape demands robust architecture that doesn't just survive, but actively thrives under immense pressure. Let's strip away the industry buzzwords to focus on the absolute essentials: **Reliability, Scalability, and Maintainability**. Let's decode these three pillars and explore how they define the systems we engineer.
\n\n
\n\n### 💥 ***Reliability: Expect the Unexpected***
\n\nWe often think of reliability in terms of uptime, but Kleppmann defines it more practically: *continuing to work correctly, even when things go wrong*. A truly resilient system anticipates faults so they don't cascade into total failures. \n\n* **Hardware Faults:** Disks crash and networks drop. We mitigate these highly probable events through hardware redundancy and software fault-tolerance.
\n* **Software Errors:** These are insidious because they are often correlated across nodes (e.g., a leap-second bug). Preventing them requires process isolation, measuring runtime behavior, and deliberately inducing faults (think Chaos Engineering) to exercise your recovery paths.
\n* **Human Errors:** We are the weakest link. The best defense is decoupling the places where people make mistakes from the places where they can cause outages—pairing safe sandbox environments with rapid rollback mechanisms.
\n\n### 📈 ***Scalability: Beyond \"Just Add More Servers\"***
\n\nScalability isn't a one-dimensional label; it's a dynamic capability to cope with increased load. But before you can scale, you must precisely *describe* your load.
\n\n* **Describing Load:** Consider Twitter’s famous fan-out challenge. Handling 12,000 tweet writes per second is relatively trivial, but delivering a celebrity's tweet to 30 million followers in under 5 seconds is an entirely different architectural beast.
\n* **Describing Performance:** Forget averages; they are deceptive. As senior architects, we must obsess over **percentiles (p95, p99, p999)**. Queueing delays (head-of-line blocking) heavily influence these tail latencies, directly impacting your most valuable users.
\n* **Coping with Load:** The eternal debate of scaling up (vertical) versus scaling out (horizontal/shared-nothing). Pragmatic architectures often blend both, built entirely around the assumptions of your specific load parameters. There is no magic scaling sauce.
\n\n### 🛠️ ***Maintainability: Avoiding the \"Big Ball of Mud\"***
\n\nIt is a well-known industry secret that the majority of software cost is heavily weighted toward ongoing maintenance, not initial development.
\n\n* **Operability:** Make life easy for your operations team. Comprehensive telemetry, good default behaviors, and predictable runtime states are non-negotiable for modern data systems.
\n* **Simplicity:** Small systems are delightful; large systems easily become a \"big ball of mud.\" Your goal is to remove *accidental complexity* by creating clean abstractions that hide implementation details.
\n* **Evolvability:** Agile is an architectural necessity, not just a management process. Your data structures and system boundaries must be plastic enough to adapt to unforeseen use cases.
\n\n### 💡 ***The Architect's Takeaway***
\n\nAs Principal Engineers, it is incredibly tempting to over-engineer for hypothetical scale. However, premature optimization often locks you into inflexible designs. Most of a system's lifecycle is spent in maintenance—prioritizing simplicity and robust operability from day one usually pays the highest dividends. You don't have to be Google or Amazon to build brilliant software; you just need to deeply understand the trade-offs of the tools in your hands.
","title":"Building for the Future: The Three Pillars of Data-Intensive Applications","tags":["distributed computing","data intensinve","performance"],"read_time":8,"created_at":"2026-06-19T12:34:52.717000Z","published":true,"summary":"Reliability, Scalability, and Maintainability","id":"DYSuqlTV2AT8ZFaeLqp7"},{"cover_image_url":"","category":"mlops","updated_at":"2026-05-28T12:34:52.717401Z","content":"# Orchestrating ML Workflows\n\nContent goes here...","tags":["airflow","mlflow","mlops"],"title":"Orchestrating ML Workflows with Airflow + MLflow","read_time":6,"created_at":"2026-05-28T12:34:52.717401Z","published":true,"summary":"End-to-end ML pipeline orchestration combining Airflow DAGs with MLflow tracking.","id":"rqUew3CJgjHKY9NNTTLz"},{"cover_image_url":"","updated_at":"2026-05-28T12:34:52.717401Z","category":"Data Engineering","content":"# ***Eventarc Advanced: Orchestrating Complex Microservices Environments***\r\n\r\n\r\n---\r\n
\r\n\r\n## Overview\r\n\r\nModern distributed applications need more than simple event routing.\r\n\r\nOrganizations increasingly require:\r\n\r\n* Centralized governance\r\n* Event filtering\r\n* Event transformation\r\n* Security controls\r\n* Observability\r\n* Multi-service orchestration\r\n\r\nTo address these needs, Google introduced **Eventarc Advanced**, a serverless eventing platform that extends Eventarc beyond basic event routing.\r\n\r\n---\r\n
\r\n\r\n## Why Eventarc Advanced?\r\n\r\nTraditional event systems often become difficult to manage as:\r\n\r\n* Services multiply\r\n* Teams grow\r\n* Event sources diversify\r\n* Governance requirements increase\r\n\r\nEventarc Advanced provides a centralized eventing layer capable of:\r\n\r\n* Ingesting events\r\n* Filtering events\r\n* Transforming payloads\r\n* Routing messages\r\n* Managing security policies\r\n* Monitoring event flow\r\n\r\nAll from a unified platform.\r\n\r\n---\r\n
\r\n\r\n## Key Capabilities\r\n\r\n### 1. Publish API\r\n\r\nAllows:\r\n\r\n* Custom applications\r\n* External systems\r\n* Third-party services\r\n\r\nto publish events using the CloudEvents format.\r\n\r\n---\r\n
\r\n\r\n### 2. Central Message Bus\r\n\r\nActs as the backbone of the event-driven architecture.\r\n\r\nBenefits:\r\n\r\n* Centralized management\r\n* Security enforcement\r\n* Observability\r\n* Flexible routing\r\n\r\nThe message bus is built on technologies such as Envoy and leverages Google Cloud networking and policy capabilities.\r\n\r\n---\r\n
\r\n\r\n### 3. Event Mediation\r\n\r\nSupports:\r\n\r\n* Real-time filtering\r\n* Event transformation\r\n* Attribute modifications\r\n* Format conversions\r\n\r\nSupported payload formats include:\r\n\r\n* JSON\r\n* Avro\r\n* Protobuf\r\n\r\nThis enables communication between systems that use different schemas or message formats.\r\n\r\n---\r\n
\r\n\r\n### 4. Reliable Delivery\r\n\r\nProvides mechanisms for:\r\n\r\n* Error handling\r\n* Retry behavior\r\n* Recovery from transient failures\r\n\r\nto improve system resilience.\r\n\r\n---\r\n
\r\n\r\n## Developer Benefits\r\n\r\nEventarc Advanced simplifies event-driven development by:\r\n\r\n* Reducing custom routing logic\r\n* Providing a unified API experience\r\n* Supporting scalable and decoupled services\r\n* Enabling event transformation without modifying applications\r\n\r\nDevelopers can focus more on business logic and less on event infrastructure.\r\n\r\n---\r\n
\r\n\r\n## Operations Benefits\r\n\r\nPlatform teams gain:\r\n\r\n* Central governance\r\n* Security controls\r\n* Monitoring\r\n* Logging\r\n* Easier troubleshooting\r\n\r\nThis reduces operational complexity across projects and teams.\r\n\r\n---\r\n
\r\n\r\n## Example: Order Processing System\r\n\r\nImagine an e-commerce platform.\r\n\r\n### Event Sources\r\n\r\n* Order Created\r\n* Payment Confirmed\r\n* Shipment Updated\r\n\r\nAll events are published to a central message bus.\r\n\r\n---\r\n
\r\n\r\n### Flow\r\n\r\n```text\r\nOrder Service\r\n |\r\n v\r\n+---\r\n
---\r\n
---\r\n
---\r\n
---\r\n
-+\r\n| Message Bus |\r\n+---\r\n
---\r\n
---\r\n
---\r\n
---\r\n
-+\r\n |\r\n +---\r\n
---\r\n
---\r\n
---\r\n
---\r\n
-+\r\n | |\r\n v v\r\nNotification Fraud Detection\r\nService Service\r\n```\r\n\r\n---\r\n
\r\n\r\n### Example Routing Rules\r\n\r\n#### New Orders\r\n\r\nCondition:\r\n\r\n```text\r\nstatus = \"new\"\r\n```\r\n\r\nAction:\r\n\r\n```text\r\nSend confirmation email\r\n```\r\n\r\n---\r\n
\r\n\r\n#### High-Value Orders\r\n\r\nCondition:\r\n\r\n```text\r\namount > 1000\r\n```\r\n\r\nAction:\r\n\r\n```text\r\nRoute to fraud detection\r\n```\r\n\r\nEvent transformations can be applied before delivery.\r\n\r\n---\r\n
\r\n\r\n## Architecture Diagram (Mermaid)\r\n\r\n```mermaid\r\ngraph TD\r\n\r\nA[Order Service]\r\nB[Payment Service]\r\nC[Shipping Service]\r\n\r\nA --> D[Message Bus]\r\nB --> D\r\nC --> D\r\n\r\nD --> E[Notification Service]\r\nD --> F[Fraud Detection]\r\nD --> G[Analytics Platform]\r\n\r\nD --> H[Cloud Run]\r\nD --> I[Cloud Functions]\r\nD --> J[External Systems]\r\n```\r\n\r\n---\r\n
\r\n\r\n## Use Cases\r\n\r\n### Large-Scale Application Integration\r\n\r\nConnect many applications and services using asynchronous communication.\r\n\r\n### AI and Analytics Pipelines\r\n\r\nFilter and transform incoming data before processing.\r\n\r\n### Hybrid and Multi-Cloud\r\n\r\nExtend event-driven workflows beyond Google Cloud into:\r\n\r\n* On-premises systems\r\n* Other cloud providers\r\n\r\n---\r\n
\r\n\r\n## Looking Ahead\r\n\r\nGoogle highlights future integration opportunities involving:\r\n\r\n* Service Extensions\r\n* Agentic applications\r\n* Security controls\r\n* AI-related services such as Model Armor\r\n\r\nThe goal is to make Eventarc Advanced a central orchestration layer for increasingly distributed and intelligent applications.\r\n\r\n---\r\n
\r\n\r\n## Key Takeaway\r\n\r\nEventarc Advanced is not simply an event router.\r\n\r\nIt introduces a centralized eventing platform that combines:\r\n\r\n* Event ingestion\r\n* Governance\r\n* Security\r\n* Filtering\r\n* Transformation\r\n* Routing\r\n* Observability\r\n\r\nmaking it easier to build and operate large-scale event-driven systems.\r\n","tags":["llm","fine-tuning","pytorch"],"title":"Orchestrating Complex Microservices Environments","read_time":11,"published":true,"created_at":"2026-05-28T12:34:52.717401Z","summary":"Connect numerous services and agents, enabling them to communicate asynchronously and reliably, even across different event formats and schemas.","id":"eHdWYceKq1R63C2GqMoj"}]