Hey - For the few brave data-interested people who've joined the channel, perhaps we could do brief intros. I'm working at a rewards and recognition SaaS company. I'm a director of Data Engineering and lead the engineering squads that are building what we refer to as our Data Platform. We're ingesting and transforming data from a multitude of sources, internal and external, and our mission is to provide comp[any-wide access to that data resource (aka. Data Democratization). We've been on our mission for around 18 months and have made good progress. We've build primarily on an AWS stack (DMS, Glue, Redshift, Quicksight) and have supplemented where it seemed to make sense (DBT, Airflow, ...). I was quite surprised on finding this community that there was so little discussion on Data as a Product or Data as a Platform... Anyone else with similar work ongoing or who'd like to intro ?

Hey Mick - that sounds pretty neat! Have you heard of Chad Sanderson? He started a Data Quality Camp community, and I bet you would find some great people to connect with on this subject there. It seems like there is somewhat of a divide between data engineering and platform/devops - to me this is strange, because who’s job is it to make sure kafka and the data streaming infra has high availability? Seems like an overlap!

As for my intro, I am a Solutions engineer responsible for onboarding/documentation for our data reliability product. Essentially we have a platform that connects to CDC and data streams, and allows you to observe, operate, and send data anywhere (like multiplexing). There is a lot more to it, but a recent focus has been experimenting with helping data engineering teams lower their warehousing costs (like with snowflake), and data security/quality. We are a AWS/GCP shop writing our services in Go, and open sourced our internal messaging CLI tool plumber. It’s a pretty neat tool, especially if you want to observe binary data or test your services.

Interesting point made there by @Jon Wright on the divide between data engineering and platform/devops. I’ve also experienced this distinction in a few of my previous work places. Tbh, both those are 2 different disciplines arguably but I also see your point on the small overlap. I guess a more broader discipline would what I call software engineering is which encompasses both disciplines. Regardless whether we work in more devops oriented roles or more in the data space, ultimately we’re working on a piece of software one way or another.

@Mick Rice I’m not a data specific engineer so not sure if it helps but I can share some relevant experience from a few of my previous companies, they’ve been doing similar thing in building an internal data platform that gets data ingested and transformed from many different sources both internal & external and then to store that data and provide it company wide. They’ve adopted a SAAS platform (Snowflake) as the central data platform at its core but also building out custom components surrounding it (by using tech like AWS Glue, Redshift, EMR, Data Pipeline, even custom Lambda functions for ETL tasks) to form an internal “framework” which makes up the internal “data platform”. feel free to PM me if you want to chat more in details…

Hey folks. Pretty excited to know we’ve got a data platform channel here. I’m an engineering manager for Platform and my focus is primarily on developer empowerment through cognitive load reduction, automation, building of self-service layers, etc. I’ve recently started developing a curiosity surrounding the topic of what data as a platform could look like. Open questions are: • What are some of the typical bottlenecks that come about from the traditional model of data engineering teams and data analytics teams which “platformisation” can optimize? • What would a self-service layer for a data engineering team that empowers development teams to independently ingest their data sets into a company’s data lake look like? Would be nice to hear the community’s thoughts on the topic

Where platform / devops meets data (science) -- https://blog.jupyter.org/devops-intelligence-3ff48a76b525

Out data platform's core is Jupyter deployed to K8s, with associated services and data lakes / puddles based on Trino, Kafka, S3, etc.

What you CAN do is use such a (business) platform for devops / SRE purposes that go beyond the capabilities of things like Grafana