[{"cover_image_url":"https://storage.googleapis.com/abatra-portfolio-assets-public/blog-img/OLTP_Systems.png","content":"### ***History | What | Why***
\n\nIn early days, a write to the database typically corresponds to a commercial transaction taking place.Making a sale, placing an order with a supplier, paying an employee’s salary, etc.\nAs databases expanded into areas that didn’t involve money changing hands, the term *transaction* nevertheless stuck, referring to a group of reads and writes that form a logical unit.
\n\nTransaction processing just means allowing clients to make low-latency reads and writes—as opposed to *batch processing jobs*, which only run periodically (for example, once per day).\nEven though databases started being used for many kinds of data—comments on blog posts, actions in a game, contacts in an address book, etc.—the basic access pattern remained similar to processing business transactions.
\n\nAn application typically looks up a small number of records by some key, using an index. Records are inserted or updated based on the user’s input. Because these applications are interactive, the access pattern became known as *online transaction processing (OLTP)*.
\n\nThese databases also started being increasingly used for data analytics, which has very different access patterns.
\nFor example, if your data is a table of sales transactions, then analytic queries might be:
\n* What was the total revenue of each of our stores in January?
\n* How many more devices than usual did we sell during our latest promotion?
\n* Which brand of baby food is most often purchased together with brand X diapers?
\n\nThese queries are often written by business analysts, and feed into reports that help the management of a company make better decisions (*business intelligence*).\nIn order to differentiate this pattern of using databases from transaction processing, it has been called *online analytic processing (OLAP)*.
\n\n
\n\n### ***OLTPs Row Based Storage Pattern***
\nTo Query this efficiently in OLTP systems indexes on `fact_sales.date_key` and/or `fact_sales.product_sk` that tell the storage engine where to find all the sales for a particular date or for a particular product.
\nBut then, a row-oriented storage engine still needs to load all of those rows (each consisting of over 100 attributes) from disk into memory, parse them, and filter out those that don’t meet the required conditions. That can take a long time.
\n\n```sql\nSELECT\n dim_date.weekday, dim_product.category,\n SUM(fact_sales.quantity) AS quantity_sold\nFROM fact_sales\nJOIN dim_date ON fact_sales.date_key = dim_date.date_key\nJOIN dim_product ON fact_sales.product_sk = dim_product.product_sk\nWHERE\n dim_date.year = 2013 AND\n dim_product.category IN ('Fresh fruit', 'Candy')\nGROUP BY\n dim_date.weekday, dim_product.category;\n```\n\n### ***Why ACID***
\n\nIn the harsh reality of data systems, many things can go wrong:
\n\n* The database software or hardware may fail at any time (including in the middle of a write operation).
\n* The application may crash at any time (including halfway through a series of operations).
\n* Interruptions in the network can unexpectedly cut off the application from the database, or one database node from another.
\n* Several clients may write to the database at the same time, overwriting each other’s changes.
\n* A client may read data that doesn’t make sense because it has only partially been updated.
\n* Race conditions between clients can cause surprising bugs.
\n\nTo be reliable, a system must deal with these faults and ensure that they don’t cause catastrophic failure of the entire system. However, implementing fault tolerance mechanisms is a lot of work.It requires a lot of careful thinking about all the things that can go wrong, and a lot of testing to ensure that the solution actually works.
\n\n### ***Transactions***
\n\nConceptually, all the reads and writes in a transaction are executed as one operation: either the entire transaction succeeds (*commit*) or it fails (*abort, rollback*). If it fails, the application can safely retry. With transactions, error handling becomes much simpler for an application.
\nBecause it doesn’t need to worry about partial failure—i.e., the case where some operations succeed and some fail (for whatever reason).
\n\n### ***Control and Locking Mechanisms***
\n\nConcurrency control is the mechanism used to ensure that multiple users or applications can interact with the database simultaneously without corrupting the data. Without concurrency control, simultaneous transactions can cause severe data anomalies, such as:
\n* *Dirty Reads:* Reading uncommitted data from another transaction that later rolls back.
\n* *Non-Repeatable Reads:* Reading the same row twice in a transaction and getting different values because another transaction updated it in between.
\n* *Phantom Reads:* Executing a query twice and seeing new \"phantom\" rows appear the second time because another transaction inserted them.
\n\nTo prevent this, relational databases primarily use *Locking Mechanisms* to serialize access to data.
\n\n### ***Lock Granularity***
\n\nDatabases don't just lock rows; they can lock data at various structural levels depending on the scope of the query.
\n* *Row-Level Locking:* Locks a single record. Excellent for concurrency (many users can edit different rows in the same table) but requires higher overhead for the database engine to manage thousands of tiny locks.
\n* *Page-Level Locking:* Locks a block of memory (usually 4KB or 8KB) that contains multiple rows. It's a middle ground between row and table locks.
\n* *Table-Level Locking:* Locks the entire table. Very low overhead, but terrible for concurrency—if a bulk update is running, no one else can read or write to that table.
\n* *Database-Level Locking:* Locks the entire database, typically only used during major structural upgrades or restores.
\n\n### ***Lock Type***
\n\n| Lock Type | Also Known As | Behaviour |\n| :--- | :--- | :--- |\n| *Shared Lock (S)* | Read Lock | Allows a transaction to read a resource. Multiple transactions can hold S-locks on the same data simultaneously. |\n| *Exclusive Lock (X)* | Write Lock | Allows a transaction to update or delete a resource. Only one transaction can hold an X-lock on data at a time. |\n
\n\n### ***ACID - Atomicity***
\n\n*Atomic* - In general, atomic refers to something that cannot be broken down into smaller parts.
\nThe system can only be in the state it was before the operation or after the operation, not something in between.
\nNote - atomicity is not about concurrency. It does not describe what happens if several processes try to access the same data at the same time, because that is covered under the letter I, for isolation.
\n\nAtomicity describes what happens if a client wants to make several writes, but a fault occurs after some of the writes have been processed.
\nSo, if the writes are grouped together into an atomic transaction, and the transaction cannot be completed (committed) due to a fault, then the transaction is aborted and the database must discard or undo any writes it has made so far in that transaction.
\n\nWithout atomicity, if an error occurs partway through making multiple changes, it’s difficult to know which changes have taken effect and which haven’t.
\nAtomicity simplifies this problem: if a transaction was aborted, the application can be sure that it didn’t change anything, so it can safely be retried.
\n\n### ***ACID - Consistency***
\n\nThe idea of ACID consistency is that you have certain statements about your data (invariants) that must always be true.
\nFor e.g., in an accounting system, credits and debits across all accounts must always be balanced.
\nHowever, this idea of consistency depends on the application’s notion of invariants, and it’s the application’s responsibility to define its transactions correctly so that they preserve consistency.
\n\nThis is not something that the database can guarantee: if you write bad data that violates your invariants, the database can’t stop you.
\nIn general, the application defines what data is valid or invalid—the database only stores it.
\nNOTE - The application may rely on the database’s atomicity and isolation properties to achieve consistency.
\n\n### ***ACID - Isolation***
\nIsolation in the sense of ACID means that concurrently executing transactions are isolated from each other: they cannot step on each other’s toes.
\nThe classic database textbooks formalize isolation as *serializability*, which means that each transaction can pretend that it is the only transaction running on the entire database.
\nThe database ensures that when the transactions have committed, the result is the same as if they had run serially (one after another), even though they may have run concurrently.
\nTypes of Isolation levels – *Read Uncommitted, Read Committed, Repeatable Reads, Serializable & Snapshot*
\n\n### ***ACID - Durability***
\nDurability is the promise that once a transaction has committed successfully, any data it has written will not be forgotten, even if there is a hardware fault or the database crashes.
\nIn a replicated database, durability may mean that the data has been successfully copied to some number of nodes.
\nIn order to provide a durability guarantee, a database must wait until these writes or replications are complete before reporting a transaction as successfully committed.
\nPerfect durability does not exist: if all your hard disks and all your backups are destroyed at the same time, there’s obviously nothing your database can do to save you.
\nIn practice, there is no one technique that can provide absolute guarantees. There are only various risk-reduction techniques, including writing to disk, replicating to remote machines, and backups—and they can and should be used together.
\n\n### ***Normalization | What | Why | Types***
\nDatabase normalization is a systematic framework for organizing data in a relational database. Its primary goal is to reduce data redundancy and improve data integrity.
\nWhile modern data warehouses and analytics engines—like BigQuery/RedShift—often favour heavily denormalized schemas (like flat tables or nested arrays) to speed up read-heavy analytical queries, the upstream transactional systems (OLTP) feeding those pipelines rely on strict normalization to maintain accuracy and prevent data corruption.
\n*WHY ? To Prevent Anomalies* – If a database is not normalized, redundant data causes structural problems known as anomalies whenever data is modified.
\nFor e.g., Update Anomaly, Insert Anomaly, Deletion Anomaly.
\n\n**Example Unnormalized Data:**
\n| EmpID | EmpName | DeptName | DeptHead | ProjectID | ProjectRole |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| 101 | Rahul | Data | Anita | P1 | Engineer |\n| 101 | Rahul | Data | Anita | P2 | Lead |\n| 102 | Sneha | Data | Anita | P1 | Engineer |\n
\n\n* *Update Anomaly* - If Anita leaves and is replaced as the Data department head, you must update multiple rows. If one row is missed, the database becomes inconsistent (Rahul and Sneha would appear to have different department heads).
\n* *Insertion Anomaly*: You cannot add a new department (e.g., \"Security\" led by \"Vikram\") until you hire an employee to put in that department, because EmpID is required for a new row.
\n* *Deletion Anomaly*: If Rahul and Sneha both finish their projects and are temporarily unassigned, deleting their rows entirely wipes out the fact that the \"Data\" department exists and is led by \"Anita\".
\n\n## ***Normal Forms***
\n* Normalization solves these anomalies by decomposing large tables into smaller, tightly focused tables linked by relationships.
\n* Normal forms are progressive rules. To achieve a higher normal form, a database must first satisfy the rules of the lower ones.
\nTypes: *1NF, 2NF, 3NF*
\n\n## ***1 NF – Unique Record & Atomic Values***
\n*The Rule:* Every column must contain atomic (indivisible) values, and each record must be unique. No repeating groups or arrays in a single column.
\n*1NF Solution:* Split the multi-valued attribute into separate rows (or a separate table, depending on the broader schema design).
\n\n**Before 1NF:**\n| EmpID | Name | Phone_Numbers |\n| :--- | :--- | :--- |\n| 101 | Rahul | 555-0101, 555-0102 |\n
\n\n**After 1NF:**\n| EmpID | Name | Phone_Number |\n| :--- | :--- | :--- |\n| 101 | Rahul | 555-0101 |\n| 101 | Rahul | 555-0102 |\n
\n\n## ***2 NF – No Partial Dependencies***
\n*The Rule:* Must be in 1NF, and all non-key attributes must be fully functionally dependent on the entire primary key. (This only applies when a table has a composite primary key).
\nConsider a table with a composite primary key of (EmpID, ProjectID):
\n* Hours depends on both the Employee and the Project.
\n* EmpName only depends on EmpID (a partial dependency).
\n* ProjectName only depends on ProjectID (a partial dependency).
\n\n*2NF Solution:* Break the partial dependencies into their own tables.
\n\n
\n\n**Table 1: Employee**\n| EmpID (PK) | EmpName |\n| :--- | :--- |\n| 101 | Rahul |\n\n**Table 2: Project**\n| ProjectID (PK) | ProjectName |\n| :--- | :--- |\n| P1 | Migration |\n| P2 | API Setup |\n\n**Table 3: Employee_Project**\n| EmpID (PK) | ProjectID (PK) | Hours |\n| :--- | :--- | :--- |\n| 101 | P1 | 40 |\n| 101 | P2 | 20 |\n\n
\n\n## ***3 NF – No Transitive Dependencies***
\n*The Rule:* Must be in 2NF, and there must be no transitive dependencies. This means non-key attributes must depend only on the primary key, and not on other non-key attributes.
\nHere, DeptLocation depends on DeptID, and DeptID depends on EmpID. Because DeptLocation relies on the primary key through another column, it's a transitive dependency. This causes the update anomaly mentioned earlier.
\n\n*3NF Solution:* Move the transitively dependent data into its own lookup table.
\n\n**Table 1: Employee**\n| EmpID (PK) | EmpName | DeptID (FK) |\n\n| 101 | Rahul | D1 |\n| 102 | Sneha | D1 |\n\n**Table 2: Department**\n| DeptID (PK) | DeptLocation |\n| :--- | :--- |\n| D1 | Pune |\n
\n\n## think. believe. grow","updated_at":"2026-07-24T12:09:16.359737Z","title":"Understanding OLTP Systems - Backbone of Transactional World & Robust Consistency","tags":["strong-consistency"," durable"," transactions"],"read_time":8,"published":true,"summary":"Online Transaction Processing (OLTP) systems - designed for fast, interactive read and write database operations","created_at":"2026-07-24T12:09:16.359737Z","category":"Data Engineering","id":"Understanding_OLTP_Systems_-_Backbone_of_Transactional_World_&_Robust_Consistency"},{"cover_image_url":"https://storage.googleapis.com/abatra-portfolio-assets-public/blog-img/3_pillars.png","content":"As engineering leaders and architects, we are constantly challenged to build **unstoppable**, high-performance systems. Today's data-driven landscape demands robust architecture that doesn't just survive, but actively thrives under immense pressure. Let's strip away the industry buzzwords to focus on the absolute essentials: **Reliability, Scalability, and Maintainability**. Let's decode these three pillars and explore how they define the systems we engineer.
\n\n
\n\n### 💥 ***Reliability: Expect the Unexpected***
\n\nWe often think of reliability in terms of uptime, but Kleppmann defines it more practically: *continuing to work correctly, even when things go wrong*. A truly resilient system anticipates faults so they don't cascade into total failures. \n\n* **Hardware Faults:** Disks crash and networks drop. We mitigate these highly probable events through hardware redundancy and software fault-tolerance.
\n* **Software Errors:** These are insidious because they are often correlated across nodes (e.g., a leap-second bug). Preventing them requires process isolation, measuring runtime behavior, and deliberately inducing faults (think Chaos Engineering) to exercise your recovery paths.
\n* **Human Errors:** We are the weakest link. The best defense is decoupling the places where people make mistakes from the places where they can cause outages—pairing safe sandbox environments with rapid rollback mechanisms.
\n\n### 📈 ***Scalability: Beyond \"Just Add More Servers\"***
\n\nScalability isn't a one-dimensional label; it's a dynamic capability to cope with increased load. But before you can scale, you must precisely *describe* your load.
\n\n* **Describing Load:** Consider Twitter’s famous fan-out challenge. Handling 12,000 tweet writes per second is relatively trivial, but delivering a celebrity's tweet to 30 million followers in under 5 seconds is an entirely different architectural beast.
\n* **Describing Performance:** Forget averages; they are deceptive. As senior architects, we must obsess over **percentiles (p95, p99, p999)**. Queueing delays (head-of-line blocking) heavily influence these tail latencies, directly impacting your most valuable users.
\n* **Coping with Load:** The eternal debate of scaling up (vertical) versus scaling out (horizontal/shared-nothing). Pragmatic architectures often blend both, built entirely around the assumptions of your specific load parameters. There is no magic scaling sauce.
\n\n### 🛠️ ***Maintainability: Avoiding the \"Big Ball of Mud\"***
\n\nIt is a well-known industry secret that the majority of software cost is heavily weighted toward ongoing maintenance, not initial development.
\n\n* **Operability:** Make life easy for your operations team. Comprehensive telemetry, good default behaviors, and predictable runtime states are non-negotiable for modern data systems.
\n* **Simplicity:** Small systems are delightful; large systems easily become a \"big ball of mud.\" Your goal is to remove *accidental complexity* by creating clean abstractions that hide implementation details.
\n* **Evolvability:** Agile is an architectural necessity, not just a management process. Your data structures and system boundaries must be plastic enough to adapt to unforeseen use cases.
\n\n### 💡 ***The Architect's Takeaway***
\n\nAs Principal Engineers, it is incredibly tempting to over-engineer for hypothetical scale. However, premature optimization often locks you into inflexible designs. Most of a system's lifecycle is spent in maintenance—prioritizing simplicity and robust operability from day one usually pays the highest dividends. You don't have to be Google or Amazon to build brilliant software; you just need to deeply understand the trade-offs of the tools in your hands.
","created_at":"2026-06-19T12:34:52.717000Z","title":"Building for the Future: The Three Pillars of Data-Intensive Applications","tags":["distributed computing","data intensinve","performance"],"read_time":8,"published":true,"summary":"Reliability, Scalability, and Maintainability","updated_at":"2026-06-19T12:34:52.717000Z","category":"data engineering","id":"DYSuqlTV2AT8ZFaeLqp7"},{"cover_image_url":"","content":"# Orchestrating ML Workflows\n\nContent goes here...","created_at":"2026-05-28T12:34:52.717401Z","title":"Orchestrating ML Workflows with Airflow + MLflow","tags":["airflow","mlflow","mlops"],"read_time":6,"published":true,"summary":"End-to-end ML pipeline orchestration combining Airflow DAGs with MLflow tracking.","updated_at":"2026-05-28T12:34:52.717401Z","category":"mlops","id":"rqUew3CJgjHKY9NNTTLz"},{"cover_image_url":"","content":"# ***Eventarc Advanced: Orchestrating Complex Microservices Environments***\r\n\r\n\r\n---\r\n
\r\n\r\n## Overview\r\n\r\nModern distributed applications need more than simple event routing.\r\n\r\nOrganizations increasingly require:\r\n\r\n* Centralized governance\r\n* Event filtering\r\n* Event transformation\r\n* Security controls\r\n* Observability\r\n* Multi-service orchestration\r\n\r\nTo address these needs, Google introduced **Eventarc Advanced**, a serverless eventing platform that extends Eventarc beyond basic event routing.\r\n\r\n---\r\n
\r\n\r\n## Why Eventarc Advanced?\r\n\r\nTraditional event systems often become difficult to manage as:\r\n\r\n* Services multiply\r\n* Teams grow\r\n* Event sources diversify\r\n* Governance requirements increase\r\n\r\nEventarc Advanced provides a centralized eventing layer capable of:\r\n\r\n* Ingesting events\r\n* Filtering events\r\n* Transforming payloads\r\n* Routing messages\r\n* Managing security policies\r\n* Monitoring event flow\r\n\r\nAll from a unified platform.\r\n\r\n---\r\n
\r\n\r\n## Key Capabilities\r\n\r\n### 1. Publish API\r\n\r\nAllows:\r\n\r\n* Custom applications\r\n* External systems\r\n* Third-party services\r\n\r\nto publish events using the CloudEvents format.\r\n\r\n---\r\n
\r\n\r\n### 2. Central Message Bus\r\n\r\nActs as the backbone of the event-driven architecture.\r\n\r\nBenefits:\r\n\r\n* Centralized management\r\n* Security enforcement\r\n* Observability\r\n* Flexible routing\r\n\r\nThe message bus is built on technologies such as Envoy and leverages Google Cloud networking and policy capabilities.\r\n\r\n---\r\n
\r\n\r\n### 3. Event Mediation\r\n\r\nSupports:\r\n\r\n* Real-time filtering\r\n* Event transformation\r\n* Attribute modifications\r\n* Format conversions\r\n\r\nSupported payload formats include:\r\n\r\n* JSON\r\n* Avro\r\n* Protobuf\r\n\r\nThis enables communication between systems that use different schemas or message formats.\r\n\r\n---\r\n
\r\n\r\n### 4. Reliable Delivery\r\n\r\nProvides mechanisms for:\r\n\r\n* Error handling\r\n* Retry behavior\r\n* Recovery from transient failures\r\n\r\nto improve system resilience.\r\n\r\n---\r\n
\r\n\r\n## Developer Benefits\r\n\r\nEventarc Advanced simplifies event-driven development by:\r\n\r\n* Reducing custom routing logic\r\n* Providing a unified API experience\r\n* Supporting scalable and decoupled services\r\n* Enabling event transformation without modifying applications\r\n\r\nDevelopers can focus more on business logic and less on event infrastructure.\r\n\r\n---\r\n
\r\n\r\n## Operations Benefits\r\n\r\nPlatform teams gain:\r\n\r\n* Central governance\r\n* Security controls\r\n* Monitoring\r\n* Logging\r\n* Easier troubleshooting\r\n\r\nThis reduces operational complexity across projects and teams.\r\n\r\n---\r\n
\r\n\r\n## Example: Order Processing System\r\n\r\nImagine an e-commerce platform.\r\n\r\n### Event Sources\r\n\r\n* Order Created\r\n* Payment Confirmed\r\n* Shipment Updated\r\n\r\nAll events are published to a central message bus.\r\n\r\n---\r\n
\r\n\r\n### Flow\r\n\r\n```text\r\nOrder Service\r\n |\r\n v\r\n+---\r\n
---\r\n
---\r\n
---\r\n
---\r\n
-+\r\n| Message Bus |\r\n+---\r\n
---\r\n
---\r\n
---\r\n
---\r\n
-+\r\n |\r\n +---\r\n
---\r\n
---\r\n
---\r\n
---\r\n
-+\r\n | |\r\n v v\r\nNotification Fraud Detection\r\nService Service\r\n```\r\n\r\n---\r\n
\r\n\r\n### Example Routing Rules\r\n\r\n#### New Orders\r\n\r\nCondition:\r\n\r\n```text\r\nstatus = \"new\"\r\n```\r\n\r\nAction:\r\n\r\n```text\r\nSend confirmation email\r\n```\r\n\r\n---\r\n
\r\n\r\n#### High-Value Orders\r\n\r\nCondition:\r\n\r\n```text\r\namount > 1000\r\n```\r\n\r\nAction:\r\n\r\n```text\r\nRoute to fraud detection\r\n```\r\n\r\nEvent transformations can be applied before delivery.\r\n\r\n---\r\n
\r\n\r\n## Architecture Diagram (Mermaid)\r\n\r\n```mermaid\r\ngraph TD\r\n\r\nA[Order Service]\r\nB[Payment Service]\r\nC[Shipping Service]\r\n\r\nA --> D[Message Bus]\r\nB --> D\r\nC --> D\r\n\r\nD --> E[Notification Service]\r\nD --> F[Fraud Detection]\r\nD --> G[Analytics Platform]\r\n\r\nD --> H[Cloud Run]\r\nD --> I[Cloud Functions]\r\nD --> J[External Systems]\r\n```\r\n\r\n---\r\n
\r\n\r\n## Use Cases\r\n\r\n### Large-Scale Application Integration\r\n\r\nConnect many applications and services using asynchronous communication.\r\n\r\n### AI and Analytics Pipelines\r\n\r\nFilter and transform incoming data before processing.\r\n\r\n### Hybrid and Multi-Cloud\r\n\r\nExtend event-driven workflows beyond Google Cloud into:\r\n\r\n* On-premises systems\r\n* Other cloud providers\r\n\r\n---\r\n
\r\n\r\n## Looking Ahead\r\n\r\nGoogle highlights future integration opportunities involving:\r\n\r\n* Service Extensions\r\n* Agentic applications\r\n* Security controls\r\n* AI-related services such as Model Armor\r\n\r\nThe goal is to make Eventarc Advanced a central orchestration layer for increasingly distributed and intelligent applications.\r\n\r\n---\r\n
\r\n\r\n## Key Takeaway\r\n\r\nEventarc Advanced is not simply an event router.\r\n\r\nIt introduces a centralized eventing platform that combines:\r\n\r\n* Event ingestion\r\n* Governance\r\n* Security\r\n* Filtering\r\n* Transformation\r\n* Routing\r\n* Observability\r\n\r\nmaking it easier to build and operate large-scale event-driven systems.\r\n","created_at":"2026-05-28T12:34:52.717401Z","title":"Orchestrating Complex Microservices Environments","tags":["llm","fine-tuning","pytorch"],"read_time":11,"published":true,"summary":"Connect numerous services and agents, enabling them to communicate asynchronously and reliably, even across different event formats and schemas.","updated_at":"2026-05-28T12:34:52.717401Z","category":"Data Engineering","id":"eHdWYceKq1R63C2GqMoj"}]