This message was deleted.

Can you share a little bit how you approach building this?

For sure! Happy to do that! I will post a little wrap-up on the approach we took to solve the problems we faced in the energy industry.

That would be super cool, I think we could all benefit from that.

Hi @Kaspar As promised, a short wrap-up on what I wrapped my head around in the energy industry regarding Data Science: The classic approach I observed at old-fashioned energy companies is a trader working with a Data Scientist who runs Excel and some VBA to provide predictions. As one can assume, there will be a lot of unversioned and untested magic Excel spreadsheets floating around the organization. Generally, the problem worsened when people started to use Python and related ML libraries to model the energy markets. Suddenly the complexity of their "scripts" drastically increased! Plus, one has to keep in mind that it is not software engineers who code these models. Overall the question is, how do we take their valuable work and transform it into a properly operatable piece of software within a reasonable amount of time?  We experimented with different ideas to solve this difficulty. One approach was to force Data Scientists to conform to specific standards before deploying their solution. This approach failed as everyone was mad at each other after some time. The business was unhappy as Data Scientists could not deliver fast, Data Scientists were unhappy with Ops and the requirements, Ops was unhappy with business of not understanding Ops. Additionally, it was always a lot of work to train new employees on topics unrelated to their domain of interest. The "throwing it over the wall" approach was a no-go to us as we knew from some other energy companies that it takes them up to 6 months to deploy the work of their Data Scientist to their production systems. Somehow, we came up with the idea to abstract away as much as possible from the Data Scientists and started to tinker with a "self-serving" approach. Conceptually, we decided that each Data Science challenge consists of a single invariant problem definition and multiple feasible solutions that one can compare against each other with a metric.  The combination of a problem, solution, and measure formed a concept we named simulation, which allowed us to disentangle Data Science challenges. Suddenly we had better control to design a system and satisfy all stakeholders. Whenever there was a new challenge, the business had to define the problem and the metric to optimize -- This was also beneficial as they had to formalize their requirements precisely. A software and data engineer then implemented this problem definition and metric on our system. Ops added the optimal pipelines to deploy the upcoming solution. For the Data Scientists, it was then super convenient to work on one of the problems. They could use our CLI to bootstrap a solution that provided a minimal running model implementation for this specific problem definition. Then they could entirely focus on their expertise in building and tweaking models. Whenever they wanted, the CLI allowed them to submit more extensive experiments to our system and easily schedule them somewhere to run as a simulation. They did not have to care about anything other than modeling.  In summary, it accelerated the development process of models as anyone could join in working on a business problem with a single CLI command. And as soon as they were happy with the performance, it was almost always guaranteed that it could be shipped automatically within minutes. I hope this gives some overview of the struggles in the energy industry.😅 Happy to answer any questions :)  Best,  Manuel PS: The technical implementation of this approach was very very prototype level. The main challenge was to design an environment/workflow everyone felt comfortable working with and which benefits the business.

This is cool, would you mind share this in #learning? @Luca Galante I think you might benefit from taking a look as well with what you are building.

For sure! @Luca Galante feel free to reach out to me if you want to discuss it in-depth 🙂