Challenges

One of the world's most geographically-distributed steel producers, with global operations and commercial presence wanted to build a single platform (data lake) to manage and drive analytics on their data.

• Data silos and disparate sources: They had 100+ TB of data scattered across a variety of sources including SQL Server, Oracle, and SAP that hindered comprehensive analysis and insights.
• Real-Time Data Requirements: The senior management recognized the need for realtime insights into inventory levels, personnel deployment, and machinery conditions. This information was crucial for optimizing resource allocation, enhancing inventory management, and proactively addressing potential equipment issues.
• Legacy Systems and Slow Analytics: The existing legacy systems were not designed to handle the volume, velocity, and variety of data generated by the company's global operations. This resulted in slow data processing, delayed insights, and an inability to respond to business needs in real time.

Searce Solution

To tackle the organization's data management challenges, our team of solvers implemented a data lake on Google Cloud, consolidating data from diverse sources and enabling real-time insights.

• Leveraging Google Cloud's robust infrastructure, Searce seamlessly integrated data from SAP, Oracle, and MS SQL Server into the data lake. This involved provisioning multiple Google Cloud services, including Dataflow, Pub/Sub, and BigQuery.
• Dataflow ETL pipelines were deployed to extract, transform, and load data from onpremises systems to Google Cloud Storage (GCS) and then again to BigQuery, creating a comprehensive data lake architecture.
• Additionally, data from IoT devices was seamlessly integrated into the data lake using Pub/Sub via an MQTT bridge. This enabled the creation of a comprehensive catalog that provided management with real-time visibility into machine workloads and perational status.

Business Outcome

• Robust Data Platform: Searce successfully addressed over twenty five use cases across seven business units, encompassing infrastructure, security, networking, data analytics, and AI/ML solutions. This breadth of impact highlights the versatility and effectiveness of the data lake platform.
• Stakeholder Engagement: Searce actively engaged with key stakeholders from the organization throughout the project. Our in-house experts conducted six workshops and four demo sessions for over fifty key personnel. This collaborative approach ensured that the data lakehouse aligned with the company's strategic goals and met the specific needs of its diverse business units.

The implementation of data lake on Google Cloud empowered the organization to transform its data management landscape, enabling real-time decision-making, optimizing resource allocation, and driving operational efficiency.