Hack Week @ Endgame
Earlier this year, Endgame hosted its annual all-hands meeting, bringing together our team from across the country for internal discussions, technical talks, and social events. This was followed by our hack week, where individuals submitted proposals, built teams, learned new skills and tackled projects together focusing on product features and workflow optimization. This year’s hack week was our most successful to date both for the solutions developed and the knowledge shared, as well as its lasting impact on encouraging cross-team collaboration and an innovative culture. Taking time away from the day-to-day pressures and deadlines may seem like a luxury, but we view it as a necessity with both business value and as a means to ensure we maintain an innovative culture and cutting-edge team as we grow. This year, we organized our hack week with a few underlying objectives in mind. I’ll walk through my perspective on the essential role of hack week for our team, and highlight some of the projects and value add to the broader team.
When organizing this year’s hack week, we had a few core objectives. These can be bucketed into people, processes, and technology. Starting with technology, we wanted to empower engineers and researchers with an opportunity for greater autonomy to choose a specific problem and develop their own solutions. We want to stay on top of the latest technologies, learn new languages, and take risks that might stray from our product roadmap. As all developers can relate, we sometimes have important tech-debt items that we want to fix but just need the time. Hack week provides this opportunity to create innovative solutions to important technical problems we face every day.
Most of the year, guidance on features and priorities comes (understandably) from product management. In contrast, hack week allows the team to focus on those advances that are directly relevant to our platform, without looking for space on the latest product roadmap. Hack week offers engineers and researchers the responsibility to propose, choose, and justify their projects, which may be outside of the immediate roadmap, customer requests, or bugs. This is one of the most liberating aspects of hack week, as it emphasizes and encourages the autonomy to independently develop proposals, collaborate with different team members, and test or learn new skills. Engineers and researchers are up-to-date on advances in malware methods and prevention techniques, and so our team keeps a list of topics and small experiments they want to try to maximize impact. For example, one team worked to stretch Endgame’s HA-CFI protection to new platforms that are difficult to monitor and detect on. If the teams are successful, these outcomes will most likely be incorporated in the next few releases of the product.
Finally, it is not simply lip service to highlight the essential role of culture in building a great company and an industry-changing product. We take this very seriously and view it as a core competitive advantage. Hack week provides a unique opportunity for collaboration between people who ordinarily don’t work together. Our team is comprised of engineers, testers, user experience, site reliability engineers, and researchers with diverse backgrounds and unique experiences from their past lives in private industry, consulting, military, and government security. Hack week is where lots of that energy and experience converge to build cool new features and share skills that will give Endgame an advantage over the competition.
Let’s take a look at some of this year’s projects to illustrate the range of projects and the big swings taken. In general, although individuals had a few weeks to brainstorm ideas and build teams, realistically most development took place over a 48 hour period. Projects addressed a range of new features or challenges, including expanding various UX, data science and tradecraft analytics. Two of these projects are described below.
At Endgame, we understand that our product will be judged by how quickly and easily we can triage problems and stop attacks. With that in mind, the team below built SMP Health, a monitoring console for our Management Platform. We have been using tools like New Relic and Splunk for monitoring, but they only loosely match our needs. This new tool pulls out precisely the performance characteristics and resource consumptions of the services we care about the most. This tool will help developers, scale engineers, and customer support. Importantly, it also offered a team member the opportunity to learn the Go language (for more on our move to Go, check out our blog on debugging Python and why this would not be needed after switching to Golang).
Chan and Nayarra of the SMP Health Team
In another project, the Streamline Models team re-vamped a slow, expensive, and labor-intensive process of generating malware machine learning (ML) models. They cut the time and expense dramatically by ruthlessly automating our malware research and development processes. We can now develop ML models faster and apply them to more problems, such as macOS malware, macros, or PDFs. This will help Endgame improve detection and prevention rates, and become even more responsive to false positives experienced by our customers.
Our Streamline Models team, comprised of data scientists, frontend engineers, and site reliability engineers
Back to Business?
One of the biggest misperceptions about hack week is that it can turn into a boondoggle, where engineers and researchers spend a week building novelty projects with no business value. Of course, without oversight and good planning, this can occur. However, from the beginning we invited corporate leadership into the planning process, into the proposal reviews, and they were part of our Shark Tank-style judging and awards ceremony at the end of the week. Bridging the gap between corporate leadership and our engineers and researchers proved essential to the success of hack week, and ensured our goals were achieved. Rather than being viewed as a break from business as usual, hack week directly supports business imperatives while advancing innovation and enhancing our culture.
Corporate leadership participated in judging the final projects that were presented to the entire company
In short, our 2018 hack week exceeded our expectations and raised the bar for future hack weeks. For others thinking about implementing a hack week, there are a few key takeaways. Focus on short, experimental projects that the team will attempt to prove as viable. Encourage taking risks and failing fast. Encourage collaboration between people who ordinarily don’t work together. Share Skills. Bring the whole company into the process for broader support and impact. We learned a lot throughout hack week, and look forward to building upon this momentum and the new connections, capabilities, and skills gained throughout the week.