https://platformengineering.org logo
#platform-culture
Title
# platform-culture
a

Andreas Lundemoen Ludviksen

03/13/2023, 7:44 AM
Treat your platform as a product - Developer support A Platform team should be run with a product mindset. Meaning the team should develop features and provide developer support at the same time. How do you organize and handle support - while making sure the platform developers don’t get too many interrupts? Do you allow developers direct contact with the platform-developers? Do you have dedicated support-people?
k

Khaled Ezzughayyar

03/13/2023, 10:17 AM
We have a dedicated on-call person every week, and create a dedicated support channel on Slack #ask-platform. Other team members can redirect requests to the on-call person if people are reaching out to them directly. We provide a FAQ, and refer to it if the question is known. The on-call person does nothing except providing support. We tried to have some sort of office-hours. But no one was interested
But a general idea is; if your platform receives many requests/questions on a specific area in the platform, it means you have a problem and your “platform” must be fixed
a

Andreas Lundemoen Ludviksen

03/13/2023, 10:41 AM
Thanks for the response Khaled! How does developers know who's on-call? We consider adding a platform-support user for direct messages as well. Some people might not be comfortable asking questions in a big group. Also you could get rid of the initial redirect. What do you think?
d

Dirk Jablonski

03/15/2023, 9:15 AM
Hi Andreas! We have a “DevOps of the Day“ (I know the ne sucks, but it's so well known already 😉). For this person, we have a Slack handle that is reassigned whenever the role changes. By this, devs can contact the DoD in out team channel, or mention them in their own ones. Works pretty well, I'd say 🙂
t

Tim Becker

03/15/2023, 8:42 PM
We also have what we call “interrupt on-call” — one person who fields the support requests, we rotate weekly. We’re at the scale where we can still get other work done too. That person watches our Slack channel. But I like the idea of reassigning the Slack handle. I’d be careful about encouraging reaching out directly for 2 reasons: 1. Easy for people to get a single go-to person they like to contact instead of whoever is on call 2. Information is lost in DMs when it could be shared and searchable in a public channel
In general, we also try and balance what we actually do when we’re supporting people. We want people to be empowered to own their own code end-to-end. If it’s a failure of the platform or a weird cloud edge case, we’ll step in and help out. But if it’s something we think they should handle, we try and point them in the right direction instead of doing it ourselves. It’s a slippery slope to becoming a “support team” where we handle EVERY cloud issue. These little nudges to encourage them to do it theirselves is an important part in bolstering DevOps culture
p

Paul Puschmann

03/21/2023, 10:53 AM
I’d be careful about encouraging reaching out directly for 2 reasons:
1. Easy for people to get a single go-to person they like to contact instead of whoever is on call
Information is lost in DMs when it could be shared and searchable in a public channel
I think this is really essential: keep all discussions problems and what not public and searchable (at least imside your company)
s

Sreehari Mohan

03/24/2023, 4:21 PM
At one of my previous firms, where I built a platform team, we started off with a Slack channel. But things got too overwhelming very quickly. It was tough to calculate SLAs and developers escalated minor issues to Directors and VPs! We then moved to JIRA Helpdesk and started responding to all Slack messages with a standard redirection message to JIRA. We also added a template on JIRA that helped them bucket their issue appropriately and derive a realistic priority for the same. We used to have an on-call person for a week too and their only responsibility was to ensure that tickets were not breaching their SLAs. This helped in preventing developers contacting the tech leads directly. We also had shadow and reverse-shadows for new on-callers and a decent set of playbooks built over time. This was even open to developers, if they wanted to solve problems by themselves
z

Zac Rosenbauer

03/28/2023, 5:42 PM
At FedEx we hired a Senior PM to oversee product, built out a bunch of processes for support requests, feature requests, bug fixes etc. Shared a roadmap with the teams and spent a lot of time in user discovery (aka what sucks, what can we do better etc.) Worked relatively well but a lot of gaps + the whole thing felt like it was held together by bubble gum and duct tape (i.e. slack workflow -> Jira -> Slack Channel for new ticket etc.)
p

Paul Puschmann

03/28/2023, 6:22 PM
… the whole thing felt like it was held together by bubble gum and duct tape …
Sometimes this might not only be a feeling, but a fact 😂 But on the other hand: if you create value for your users, this can be fine at first. It is then the task to bring more stability into this setup.
z

Zac Rosenbauer

03/28/2023, 6:38 PM
How’d you do that or how have you seen that accomplished?
p

Paul Puschmann

03/28/2023, 7:02 PM
Well this depends heavily on the topic, but starting with some sort of MVP and then improving this to a real was “the way”. Example: we had some custom container-environment (built in Docker Swarm) where we wanted to improve the visibility to our developers. So we started with some small go-application that scraped the Docker-Swarm API and displayed the status in a nicely formatted table. Information: service-name, actual instance-count / desired instance count When we migrated to Nomad the information of the orchestrator was added and also the memory-limits, memory status. Also added over time: • authorization for the “restart”-buttons of each container. • deep-links to Grafana • deep-links to ELK • deep-links to Instana • …
z

Zac Rosenbauer

03/28/2023, 7:33 PM
Ahhhh Im talking less about the tech more about everything surrounding it, docs, support requests, logging bugs/features etc. We tried (to some degree of success) to treat our product teams like external customers aka customer support, docs, roadmapping etc was SUPER important.
p

Paul Puschmann

03/28/2023, 7:59 PM
We tried (to some degree of success) to treat our product teams like external customers aka customer support, docs, roadmapping etc was SUPER important.
Yes, this helps. At least in our perspective the direct user feedback was most important. If possible: dog-fooding It helped a lot to have direct contact to the devs (users) and have a good benevolent connection.
z

Zac Rosenbauer

03/28/2023, 8:00 PM
❤️ on the Dog-fooding
j

Jesse Stafford

03/28/2023, 9:43 PM
I love the idea of treating the product teams as customers and showing roadmaps!
z

Zac Rosenbauer

03/28/2023, 9:43 PM
Jesse I DM'd you. I'd love to chat ☝️
o

Onno Ceelen

03/31/2023, 7:27 PM
We have a small, dedicated 1st-line support team with a support ticket solution. Works like a charm 👏. Our product teams are very happy with this setup. Before, our platform teams were supporting ad-hoc via chats solutions, resulting in a ticket-driven-platform, high cognitive load, disruptions and delivery predictability issues. The ticket solution also gives us insights into data, so we have a better understanding on what works and what doesn’t work. Very helpful for product-led, evidence-driven internal platforms.
z

Zac Rosenbauer

03/31/2023, 11:01 PM
@Onno Ceelen would you be up to chat on a zoom? I'd love to see how you are doing it
u

Umang Hain

05/03/2023, 2:01 PM
Our approach is aligned with what’s mentioned by most people above. Each of our component teams making up the platform say networking, persistence, build systems etc have dedicated on call support. The team members on support rotate every day. Most teams have their
product-user
slack channel so the teams know which channel to go to ask the questions. Users can ask the
@product-interrupt
for help. Previously we were very heavy on those interrupts which alluded to product and documentation gaps. We then fixed those gaps and eventually moved to the model where only provide support 3 days a week (unless something is burning 🔥 ). As we ACK the ask, we have Slack workflows set up which creates JIRA tickets in their projects and label them as interrupts. That way we can find the issues where interrupts had to spend time and ask what actions were taken to prevent them from happening again. Interrupts, on those days do not work on backlog. If they have time, they fix these gaps in documentation to prevent these from popping up again.
m

Marcos Lobo

05/12/2023, 10:54 PM
@Sreehari Mohan about redirecting developers to the Jira Helpdesk. Did it work better? We're de developers more satisfied?
s

Sreehari Mohan

05/14/2023, 8:25 AM
Not really @Marcos Lobo as developers felt this was a redundant step but this was a mandatory first step for us as it helped in understanding the major problems we were facing instead of catering to only the loudest voices
117 Views