SDE-1 | Gemscap (Trading Firm) | YOE-1 | Rejected

My professional background includes working as a Quant and Algo Developer at Futures First, followed by a software developer role at TCS. My interview at Gemscap presented a significant system design challenge: how to manage millions of incoming WebSocket data streams for numerous subscribed traders within an on-premises trading firm, crucially without relying on cloud services like AWS. I was tasked with outlining a complete system architecture. My proposed approach involved a connection management service, a service for managing the raw data stream without immediate processing, and then a message queue for FIFO delivery to traders. The interviewer challenged my initial assumption about creating a separate queue for each subscriber, suggesting it wasn't an optimal design. I then elaborated on a layer for processing messages, such as converting news data into a specific format or adding timestamps, and a broadcasting service for delivery to many traders. I clarified that a subscription manager service would act as a traffic controller. The interviewer guided me to acknowledge this design as a 'fan-out' architecture, which I then proceeded to explain further. The second question focused on efficient strategies for storing both live and old news data. I suggested a cold/hot storage approach, but the interviewer implied using Array Buffers for efficient data transfer, explaining their utility with WebSockets for binary messages.