> Big screen TVs occupy a lot of warehouse space and have very low margins. ... Discounting TVs to free up space for higher margin or faster selling products can significantly impact the bottom line.
That's fascinating. Very entertaining dive into the complexity of seemingly ordinary stuff. Thanks for sharing.
> I cannot tell you how much of a pain building that data pipeline was.
Did the project survive after all? Or did it slowly die on crutches? :D
It survived and is still running. As an outside consultant, I have the luxury of saying and doing things that will get an internal employee fired. I also had the US COO and CTO as backers.
I'm curious if you pulled off building a clean solution or if all the complexity created a pile of tech debt. Meaning the pipeline is running, yet it's expensive to maintain it.
> It will be interesting to see how the next 18 months treat our currently oversized data tools landscape. There's not enough space for all the players we now have.
There is so much duplication (5-10 tools that do the same thing) and too many tools that don't fit a data science life cycle. The latter is filled with train/test/deploy tools that are easily replaced with self-service solutions.
I follow it closely because I need to advise clients about infrastructure and help data teams develop the infrastructure roadmap. I stop at the point of being a data engineer or the person maintaining the infrastructure.
I was hoping to see a different answer. :D Is there a general tooling baseline that you could share? I'm lost on how to start monitoring and evaluating everything to confidently advise something. It feels like you need to be sure your suggestion is the best for the specific need, not simply okay. Potential maturity progressions add even more complexity, as I can see it.
Learn the ecosystems and you'll have a better grasp on how each solution meets current needs and can scale to meet future needs. IBM, SAP, Salesforce, and Microsoft (obviously more) are the core players to start with. Those companies all provide someone to help guide you through the decision process and educate you on their solutions. Lean on them for a deeper dive into ecosystem fit and supporting resources to make a final recommendation.
For the final decision, yes. To learn, start with the large vendors and read through their solution ecosystems. Focus on implementations and content from people who have deployed those ecosystems.
This helps. By this time, it clicked that I had overthought it and gotten to it with the wrong perception of the need to give a recommendation on the spot. It still requires research for the final decision. The difference will lie in the ability to conclude quicker. However, prior learning couldn't hurt. Thanks!
> Big screen TVs occupy a lot of warehouse space and have very low margins. ... Discounting TVs to free up space for higher margin or faster selling products can significantly impact the bottom line.
That's fascinating. Very entertaining dive into the complexity of seemingly ordinary stuff. Thanks for sharing.
> I cannot tell you how much of a pain building that data pipeline was.
Did the project survive after all? Or did it slowly die on crutches? :D
It survived and is still running. As an outside consultant, I have the luxury of saying and doing things that will get an internal employee fired. I also had the US COO and CTO as backers.
I'm curious if you pulled off building a clean solution or if all the complexity created a pile of tech debt. Meaning the pipeline is running, yet it's expensive to maintain it.
The first pipeline was messy and we needed to rebuilt it a year later once we had developed a reference implementation. That took several iterations.
> It will be interesting to see how the next 18 months treat our currently oversized data tools landscape. There's not enough space for all the players we now have.
I'm not sure I get why there's not enough space
There is so much duplication (5-10 tools that do the same thing) and too many tools that don't fit a data science life cycle. The latter is filled with train/test/deploy tools that are easily replaced with self-service solutions.
How closely do you follow the data stack landscape as a strategist? What do you look for? Where do you stop?
I follow it closely because I need to advise clients about infrastructure and help data teams develop the infrastructure roadmap. I stop at the point of being a data engineer or the person maintaining the infrastructure.
I was hoping to see a different answer. :D Is there a general tooling baseline that you could share? I'm lost on how to start monitoring and evaluating everything to confidently advise something. It feels like you need to be sure your suggestion is the best for the specific need, not simply okay. Potential maturity progressions add even more complexity, as I can see it.
Learn the ecosystems and you'll have a better grasp on how each solution meets current needs and can scale to meet future needs. IBM, SAP, Salesforce, and Microsoft (obviously more) are the core players to start with. Those companies all provide someone to help guide you through the decision process and educate you on their solutions. Lean on them for a deeper dive into ecosystem fit and supporting resources to make a final recommendation.
So the suggestion is to start a conversation with some big vendors and their offers to a particular company you work with. Did I get it right?
For the final decision, yes. To learn, start with the large vendors and read through their solution ecosystems. Focus on implementations and content from people who have deployed those ecosystems.
This helps. By this time, it clicked that I had overthought it and gotten to it with the wrong perception of the need to give a recommendation on the spot. It still requires research for the final decision. The difference will lie in the ability to conclude quicker. However, prior learning couldn't hurt. Thanks!
Offtopic: I just stumbled upon an example of a book cover related to data, which delivers a message visually, just for inspiration. https://www.linkedin.com/posts/allendowney_we-have-a-cover-what-do-you-think-https-activity-7051569151059849218-hUPo?utm_source=share&utm_medium=member_desktop
This was a challenging read for me. I'd handle it better if it was in 2 pieces.
It got more comfortable when I looked at the graphics, though: https://www.sap.com/dam/application/shared/graphics/sap-datasphere-marketecture-graphic.svg
I'll drop a couple of questions separately to make answering them easier