Enterprise Integration
Integration Styles & Foundations
Batch file exchange between systems via shared file system or FTP/SFTP. Asynchronous, decoupled, simple. Large data volumes efficiently transferred. One system writes file, another reads. Common formats: CSV, XML, JSON, EDI. File naming conventions with timestamps. File processing acknowledgment. Polling directory for new files. Scheduled batch processing (nightly, hourly). Simple but lacks real-time capability. Mainframe-to-cloud migrations often start here.
Multiple applications access common database. Tight coupling through shared schema. Immediate data availability. No network calls between apps. Schema changes affect all systems. Concurrency and locking concerns. Encapsulation violated (direct data access). Performance bottleneck on database. Legacy integration pattern. Difficult to version independently. Prefer for tightly coupled modules, not distributed systems. Often anti-pattern in microservices.
Synchronous request-response communication. Calling remote method appears local. gRPC, Apache Thrift, SOAP, XML-RPC. Strong typing with protocol buffers or WSDL. HTTP/2 for gRPC performance. Blocking calls, caller waits for response. Tight temporal coupling. Network failures appear as method failures. Timeouts essential. Service meshes for observability. Client and server tightly coupled on interface. Suitable for low-latency internal services.
Asynchronous event-driven communication via message broker. Producers send, consumers receive independently. Temporal decoupling (systems don't need to be up simultaneously). Message queues (point-to-point) or topics (pub/sub). RabbitMQ, Apache Kafka, AWS SQS/SNS, Azure Service Bus. At-least-once, at-most-once, exactly-once semantics. Enables event-driven architectures. Scalable and resilient. Message ordering considerations. Eventual consistency patterns.
Choosing integration approach based on requirements. File transfer: batch, large volumes, legacy systems. Shared database: tight coupling acceptable, single team. RPC: synchronous, low latency, request-response. Messaging: asynchronous, decoupled, event-driven. Consider: latency needs, coupling tolerance, scalability, failure handling. Hybrid approaches common (API for queries, events for updates). RESTful HTTP for external, messaging for internal. Saga pattern combines sync and async.
Combining synchronous and asynchronous patterns. API request triggers, async processing via queue, callback on completion. Command via API, events via messaging. CQRS: sync commands, async query model updates. Request-acknowledge-callback pattern. Webhook for async notifications. Long-running operations: accept request (202), poll status endpoint, push notification when done. Balances user experience (sync) with scalability (async). Saga pattern spans sync/async boundaries.
Message Routing Patterns
Route messages based on content inspection. Examine message body, headers, or properties. Conditional routing logic (if-then-else). Different destinations based on message type, priority, customer ID. Apache Camel Content-Based Router. AWS EventBridge rules. Spring Integration router. XPath or JsonPath for message evaluation. Flexible but requires message parsing. Dynamic routing without changing producers.
Selective message forwarding based on criteria. Discard messages not meeting conditions. Filter by message type, priority, age. Reduce downstream load. Event filtering in event-driven architectures. Kafka consumer filters. AWS EventBridge filtering. Declarative filter expressions (SQL-like). Can filter at source (publisher) or destination (subscriber). Prevents processing unwanted messages. Complementary to routing.
Runtime routing table updates without code changes. Control channel for routing rules. Route messages based on current system state. Adaptive routing during failures or load. Feature flags to route to different services. A/B testing with traffic splitting. Service registry integration. Database-driven routing rules. Allows operational routing changes. Circuit breaker integration. Blue-green/canary routing at message level.
Breaking composite messages into individual parts. Single message with multiple items split into separate messages. Batch file becomes individual records. Aggregate message into component messages. Each part processed independently. Enables parallel processing. Scatter-gather pattern uses splitter. Original message ID preserved in parts for correlation. Apache Camel Splitter. AWS Step Functions Map state. Inverse of Aggregator pattern.
Combining multiple messages into one. Collect related messages (correlation ID). Wait for all parts or timeout. Emit single combined message. Scatter-gather response aggregation. Join results from multiple services. Stateful operation (requires message store). AWS Step Functions, Apache Camel Aggregator. Completion strategies: all messages, timeout, condition. Shopping cart: aggregate items. Inverse of Splitter.
Message carries its own routing instructions. Itinerary attached to message. Each processor reads next step and forwards. Dynamic workflow without centralized orchestrator. Step functions as data, not code. Loan processing through multiple approval steps. Each service determines next hop. Apache Camel Routing Slip. Flexible but complex debugging. Alternative to orchestration. Choreography with explicit routing.
Message Transformation Patterns
Field-level transformation and mapping between formats. Source schema to target schema. Handles renaming, type conversion, restructuring. Visual mapping tools (Mulesoft DataWeave, XSLT). Declarative transformations. Nested object mapping. Array transformations. Expression language for logic. Version management for mappings. Schema evolution handling. Enterprise integration tools provide mappers. Can be code or configuration.
Enterprise-wide common message format. All services translate to/from canonical model. N integrations instead of N². Reduces point-to-point complexity. Common vocabulary and semantics. Schema management challenge. Services maintain adapters to canonical. Versioning canonical model difficult. Can become enterprise bottleneck. XML Schema, JSON Schema, Protobuf for canonical. Alternative: context-specific models with anti-corruption layers.
Converting between system-specific formats. Adapter pattern for messaging. Each system speaks own language, translator converts. XML to JSON, SOAP to REST, Protobuf to JSON. Protocol conversion. Legacy system integration. Translation layer between bounded contexts. Apache Camel Type Converter. AWS EventBridge transformers. Can be stateless or stateful. Often combined with routing.
Adding metadata without modifying payload. Wrap message with routing info, security context, tracing headers. Original message preserved. Headers for routing, filtering. Correlation ID, timestamp, source. JWT token in wrapper. CloudEvents specification. Kafka message key and headers. Enables cross-cutting concerns without payload changes. Unwrap at destination. Layer 7 envelope for layer 7 payload.
Augmenting messages with additional data. Lookup related data from external sources. Add customer details to order message. Geocode address in shipping message. Enrich with real-time data (stock price, weather). Call external API synchronously. Query database for reference data. Cache for performance. Stateless transformation. Apache Camel Content Enricher. Keeps publishers simple. Risk: enrichment failure blocks processing.
Storing large payloads externally with reference tokens. Message contains claim check (ID or URL). Receiver retrieves payload using claim check. Reduces message size in broker. Keeps queues lightweight. Store in S3, blob storage, database. Claim check expires after TTL. Useful for images, videos, large documents. Decouples message transport from payload storage. Kafka with large messages uses this. Trade-off: extra hop to retrieve.
ESB & Integration Platforms
Centralized integration infrastructure. Hub-and-spoke or bus topology. Message mediation, routing, transformation. Protocol conversion (SOAP-REST-JMS). Service orchestration for workflows. Centralized governance and monitoring. Traditional integration middleware. Mule ESB, Oracle Service Bus, IBM Integration Bus. Can become monolithic bottleneck. Contrast with decentralized microservices. Modern: lightweight integration, avoid ESB anti-patterns. Useful for legacy integration.
Protocol conversion, transformation, and routing in ESB. Adapters for various protocols (HTTP, JMS, FTP, SMTP). Message transformation (XML, JSON, EDI, CSV). Content-based routing logic. Header-based routing. Mediation sequences/flows. WSO2 ESB mediation. Camel routes. Centralized mediation vs adapter per service. Can add latency. Alternative: smart endpoints, dumb pipes (microservices philosophy).
Coordinating multi-step workflows centrally. BPEL, BPMN for orchestration. Central orchestrator calls services. Knows workflow logic. Synchronous or asynchronous steps. Saga orchestration. AWS Step Functions, Azure Logic Apps, Camunda. Human tasks in workflows. Compensation for failures. Visibility into process state. Contrast with choreography (decentralized). Orchestration good for complex workflows. Trade-off: central point of failure.
Cloud-based integration services. MuleSoft Anypoint, Dell Boomi, Informatica Cloud, Workato, Zapier. Pre-built connectors for SaaS applications (Salesforce, Slack, etc.). Low-code/no-code integration. Hybrid integration (cloud and on-premises). API management included. Visual flow designers. Scalable cloud infrastructure. Multi-tenant. Reduced operational overhead. Modern alternative to on-premises ESB. Subscription pricing.
Gateway patterns for ESB integration. API Gateway fronts ESB services. Rate limiting, authentication, caching. Developer portal for APIs. API composition (Backend for Frontend pattern). ESB handles backend integration, gateway handles external. Apigee, Kong, AWS API Gateway + ESB. API versioning and lifecycle. Analytics and monitoring. Security policies. Modern integration: API-led connectivity. Three-tier: system, process, experience APIs.
Decentralized integration patterns vs centralized ESB. Microservices: smart endpoints, dumb pipes. Each service owns integration logic. Lightweight messaging (HTTP, gRPC, messaging). No central ESB bottleneck. Service mesh for cross-cutting concerns. API gateway for edge. Event-driven with Kafka. Trade-offs: ESB provides governance, visibility, but less agile. Microservices more agile but require discipline. Hybrid: use ESB for legacy, microservices for new.
Message Endpoint Patterns
Periodic message retrieval from queue. Consumer checks for messages at intervals. Pull model. Simple implementation. Control consumption rate. Works with REST APIs (long polling). SQS polling, database polling. Configurable poll interval. Inefficient for low message rates. Delay between message arrival and processing. Good for batch processing. Spring Integration poller. Can miss messages if not careful.
Push-based message delivery. Broker pushes messages to consumer. No polling overhead. Immediate message processing. Callback or listener pattern. RabbitMQ push, Kafka consumer poll (but push-like). WebSocket, Server-Sent Events for browser. AWS Lambda triggers. Low latency. Requires consumer to be always available. Backpressure handling important. Reactive streams. Prefetch and buffering. Acknowledges messages after processing.
Load distribution across multiple consumers. Multiple instances consume from single queue. Message delivered to one consumer only. Horizontal scaling of message processing. RabbitMQ work queues, SQS multiple consumers. Partition-based in Kafka (one consumer per partition). No message duplication (within partition). Increases throughput. Failure of one consumer doesn't stop processing. Consumer groups. Auto-scaling based on queue depth.
Distributing work to specialized processors. Single consumer receives, dispatches to handlers. Handler per message type or command. Command pattern implementation. Spring @MessageMapping annotations. Switch statement anti-pattern vs strategy. Polymorphic dispatch. Actor model dispatching. Reduces code duplication. Central routing logic. Extensible for new message types. Alternative: separate queues per type.
Message filtering at consumer level. Consumer specifies filter criteria. JMS message selectors. Kafka consumer filters (client-side). SQL-like expressions for filtering. Subscribe to subset of messages. Reduces unnecessary message delivery. Can be inefficient (broker still sends). Better: broker-side filtering or topic design. Use when consumer needs dynamic filtering. Combine with message filter pattern.
Handling duplicate message delivery safely. At-least-once delivery causes duplicates. Detect and skip duplicate processing. Deduplication using message ID. Store processed IDs (database, cache). Time-window deduplication. Idempotent operations naturally safe (update operations). Non-idempotent needs explicit handling (payment processing). Kafka exactly-once with transactions. SQS message deduplication. Essential for reliable messaging. Idempotency keys in APIs.
Reliability & Error Handling
Ensuring message delivery semantics. At-least-once: message delivered one or more times (duplicates possible). At-most-once: message delivered zero or one time (loss possible). Exactly-once: delivered exactly once (hard to achieve, requires coordination). Kafka transactions for exactly-once. RabbitMQ publisher confirms and consumer acknowledgments. Message persistence to disk. Delivery guarantees vs performance trade-offs. Idempotent receivers for at-least-once.
Handling permanently failed messages. Messages that can't be processed after retries. Poison messages moved to DLQ. Prevents blocking main queue. Investigate and replay manually. SQS DLQ, RabbitMQ dead letter exchange. Alerts on DLQ depth. Separate processing for DLQ. Forensics and debugging. Expiration policies for DLQ. Prevents message loss. Alternative: dead letter topic. Monitor DLQ metrics.
Handling transient failures. Immediate retry, exponential backoff, jitter. Retry count limits. Transient vs permanent failures. Circuit breaker integration (stop retrying if open). Retry storm prevention. Idempotent operations required. RabbitMQ delayed retry with TTL and DLX. SQS visibility timeout. Polly library (.NET). Resilience4j (Java). Backoff strategies. Max retry attempts. Fallback after retries exhausted.
Reversing completed operations in distributed systems. Saga pattern compensating transactions. Forward recovery not possible, compensate instead. Cancel order, refund payment, release inventory. Idempotent compensations. Compensation order (reverse of execution). Orchestrated vs choreographed sagas. Workflow engines track compensation. Business process compensation. Not ACID transactions. Eventual consistency. Compensating delete (soft delete).
Time-to-live and stale message handling. Messages with TTL expire if not processed. Prevent processing old data. RabbitMQ message TTL, SQS message retention. Use for time-sensitive data (stock quotes, notifications). Expired messages moved to DLQ or discarded. Per-message or per-queue TTL. Timestamp in message for age calculation. Consumer checks message age. Prevents resource waste on stale data. Important for real-time systems.
Ensuring delivery across network failures. Sender stores message until broker acknowledges. Persistent local queue at sender. Transactional outbox pattern. Message stored in database with transaction. Background process forwards to broker. Eventual delivery guarantee. Survives sender crashes. IBM MQ assured delivery. Kafka producer acks. Trade-off: complexity and latency. Essential for critical messages (payments, orders). Alternative to synchronous delivery.
