Category: Delphix

Applications are the nexus of the modern enterprise. They simplify operations, speed execution, and drive competitive advantage. Accelerating the application lifecycle means accelerating the business. Increasingly, organizations turn to public and private clouds, SaaS offerings, and outsourcing to hasten development and reduce risk, only to find themselves held hostage by their data.

Applications are nothing without data. Enterprise applications have data anchored in infrastructure, tied down by production requirements, legacy architecture, and security regulations. But projects demand fresh data under the control of their developers and testers, requiring processes to work around these impediments. The suboptimal result leads to cost overruns, schedule delays, and poor quality.

Agile development requires agile data. Agile data empowers developers and testers to control their data on their schedule. It unburdens IT by efficiently providing data where it is needed independent of underlying infrastructure. And it accelerates application delivery by providing fresh and complete data whenever necessary. It grants its users super powers.

Many technologies can solve part of the agile data problem, but a partial solution still leaves you with suboptimal processes that impede your business. A complete agile data solution must embrace the following attributes.

Non Disruptive Synchronization

Production data is sensitive. The environment has been highly optimized and secured, and its continued operation is critical to the success of the business – introducing risk is unacceptable. An agile data solution must automatically synchronize with production data such that it can provide fresh and relevant data copies, but it cannot mandate changes to how the production environment is managed, nor can its operation jeopardize the performance or success of business critical activities.

Service Provisioning

Data is more than just a sequence of bits. Projects access data through relational databases, NoSQL databases, REST APIs, or other APIs. An agile data solution must move beyond copying the physical representation of the data by instantiating and configuring the systems to access that data. Leaving this process to the end users induces delays and amplifies risk.

Source Freedom

Data is pervasive. Efforts to mandate a single data representation, be it a particular relational or NoSQL system, rarely succeed and limit the ability of projects to choose the data representation most appropriate for their needs. As projects needs diversify the data landscape, the ability to manage all data through a single experience becomes essential. This unified agile experience necessitates a solution not tied to a single data source.

Platform Independence

The premier storage, virtualization, and compute platforms of today may be next year’s legacy architecture. Solutions limited to a single platform inhibit the ability of organizations to capitalize on advances in the industry, be it a high performance flash array or new private cloud software. Agility over time requires a solution that is not tied to the implementation of a particular hardware or software platform.

Efficient Copies

Storage costs money, and time costs the business. Agile development requires a proliferation of data copies for each developer and tester, magnifying these effects. Working around the issue with partial data leads to costly errors that are caught late in the application lifecycle, if at all. An agile solution must be able to create, refresh, and rollback copies of production data in minutes while consuming a fraction of the space required for a full copy.

Workflow Customization

Each development environment has its own application lifecycle workflow. Data may need to be masked, projects may need multiple branches with different schemas, or developers may need to restart services as data is refreshed. Pushing responsibility to the end user is error prone and impedes application delivery. An agile solution must provide stable interfaces for automation and customization such that it can adapt to any development workflow.

Self Service Data

Developers and testers dictate the pace of their assigned tasks, and each task affects the data. Agile development mandates that developers have the ability to transform, refresh, and roll back their data without interference. This experience should shield the user from the implementation details of the environment to limit confusion and reduce opportunity for error.

Resource Management

Each data copy consumes resources through storage, memory, and compute. Once developers experience the power of agile data, they will want more \copies, run workloads on them for which they were not designed, and forget to delete them when they are through. As these resources become scarce, the failure modes (such as poor performance) become more expensive to diagnose and repair. Combatting this data sprawl requires visibility into performance and capacity usage, accountability through auditing and reports, and proactive resource constraints.

Delphix is the agile data platform of the future. You can sync to your production database, instantly provision virtual databases where they are needed using miniscule amount space, and provide each developer their own copy of the data that can be refreshed and rolled back on demand. This platform will become only more powerful over time as we add new data sources, provide richer workflows targeting specific applications and use cases, and streamline the self service model. An enterprise data strategy without Delphix is just a path to more data anchors, necessitating suboptimal processes that continue to slow application development and your business.

At Delphix, we just concluded one of our recurring Engineering Kickoff events where we get everyone together for a few days of collaboration, discussion, idea sharing, and fun. In this case it included, for the first time, an all-day hackathon event. To be honest, it was a bit of an experiment and one where we were unsure of how it would be received. We had all read about, participated in, or hear praise of, hackathons at other companies, but these companies were always more consumer-focused or had technologies that were more easily assembled into different creations. As an enterprise software company, we were concerned that even the simplest projects would be too complex to turn around over the course of a day. Given the potential benefit, however, it was clearly something we wanted to experiment with – any failure would also be a learning opportunity.

Some companies go big or go home when it comes to hackathons – week long activities, physical hacks, etc. We wanted to preserve freedom but be a little more targeted. The directive was simple: spend a day doing something unrelated to your normal day job that in some way connects to the business. People volunteered ideas and mentorship ahead of time so that even the newest engineers could meaningfully participate. The result was a resounding success. Whether people were able to give a demo, sketch on a whiteboard, or just speak to their ideas and the challenges they faced, everyone pushed themselves in new directions and walked away having learned something through the experience.

The set of activities covered a wide swath of engineering, including:

• Using D3.js for visualizing analytics data
• “zero copy” iSCSI in illumos
• web portal for customer data analysis
• “zpool dump” to store pool metadata for offline zdb(1M) use
• Real time engineering dashboard to aggregate commits, bugs, reviews, and more
• “D++” DTrace syntactic sugar: function elapsed time, unrolled while loops, callers array
• Mobile application to monitor Delphix alerts and faults
• Global symbol tab completion for MDB
• Network performance tool
• Speeding up unit tests
• Browser usage analytics
• ‘zfs send’ to a POSIX filesystem
• BTrace++ (a.k.a. CATrace) to make java tracing safe and easy
• New V2P (virtual to physical) mechanisms in Delphix
• Tools to more easily deploy changes to VMs

For myself, I put together a prototype of a hosted SSH/HTTP proxy for use by our support organization. This was my first real foray into the world of true PaaS cloud software – running node.js, redis, and cloudAMQP in a heroku instance, and it’s been incredibly interesting to finally play with all these tools I’ve read about but never had a reason to use. I will post details (and hopefully code) once I get it into slightly better shape.

Only a fraction of these are really what I would consider a contribution to the product itself, which is where our initial trepidation around a hackathon went awry. No matter how complex your product or how high the barriers to entry , engineers will find a way to build cool things and try out new ideas in a hackathon setting. Everything that people did, from learning how to make changes to our OS to improving our quality of life as engineers to testing new product ideas, will provide real value to the engineering organization. On top of that, it was incredibly fun and a great way to get everyone working together in different ways.

It’s something we’ll certainly look at doing again, and I’d recommend that every company, organization, or group, find some way to get engineers together with the express purpose of working on ideas not directly related to their regular work. You’ll end up with some cool ideas and prototypes, and everyone will learn new things while having fun doing it.

The other week I had a particularly disheartening discussion with a potential new hire. I typically describe our engineering organization at Delphix as a bottoms-up meritocracy where smart people seize opportunities to impact the company through through thoughtful execution and data-driven methodology (a.k.a. buzzword bingo gold). In this case, after hours of discussions, I couldn’t shake this engineer’s fixation with understanding how his title would affect his ability to have impact at Delphix. After deciding that it was not a good cultural fit, I spent some time thinking about what defines our engineering culture and what exactly it was that I felt was such a mismatch. Rather than writing some pseudo-recruiting material extolling the virtues of Delphix, I thought I’d take a cue from Bryan’s presentation on corporate open source anti-patterns (video) and instead  look at some engineering cultural anti-patterns that I’ve encountered in the past. What follows is a random collection of cultural engineering pathologies that I’ve observed in the past and have worked to eschew at Delphix.

The Thinker

This engineer believes his responsibility is to think up new ideas, while others actually execute those ideas. While there are certainly execution-oriented engineers with an architect title out there that do great work, at Delphix we intentionally don’t have such a title because it can send the wrong message: that “architecting” is somehow separate from executing, and permission to architect is something to be given as a reward. The hardest part of engineering comes through execution – plotting an achievable path through a maze of code and possible deliverables while maintaining a deep understanding of the customer problem and constraints of the system. It’s important to have people who can think big, deeply understand customer needs, and build complex architectures, but without a tangible connection to the code and engineering execution those ideas are rarely actionable.

The Talker

Often coinciding with “The Thinker”, this engineer would rather talk in perpetuity rather than sit down and do actual work. Talkers will give plenty of excuses about why they can’t get something done, but the majority of the time those problems could be solved simply by standing up and doing things themselves. Even more annoying is their penchant for refusing to concede any argument, resulting in orders of magnitude more verbiage with each ensuing email, despite attempts to bring the discussion to a close. In the worst case the talker will provide tacit agreement publicly but fume privately for inordinate amounts of time. In many cases the sum total of time spent talking about the problem exceeds the time it would take to simply fix the underlying issue.

The Entitled

This engineer believes that titles are granted to individuals in order to expand her influence in the company; that being a Senior Staff Engineer enables her to do something that cannot be accomplished as a Staff Engineer. Titles should be a reflection of the impact already achieved through hard work, not a license granted by a benevolent management. When someone is promoted, the reasons should be obvious to the organization as a whole, not a stroke of luck or the result of clever political maneuvering. Leadership is something earned by gaining the respect of your peers through execution, and people who would use their title to make up for a lack of execution and respect of their peers can do an incredible amount of damage within an enabling culture.

The Owner

This engineer believes that the path to greater impact in the organization is through “owning” ideas and swaths of technology. While ownership of execution is key to any successful engineering organization (clear responsibility and accountability are a necessity), ownership of ideas is toxic. This can lead to passive-aggressive counter-productive acts by senior engineering, and an environment where junior engineers struggle to get their ideas heard. The owner rarely takes code review comments well, bullies colleagues that encroach on her territory, and generally holds parts of the product hostage to her tirades. Metastasized in middle management, this leads to ever growing fiefdoms where technical decisions are made for entirely wrong organizational reasons. Ideas and innovation come from everywhere, and while different parts of the organization are better suited to execution of large projects based on their area of expertise, no one should be forbidden from articulating their ideas due to arbitrary assignment.

The Recluse

Also known as “the specialist”, this engineer defines his role in the most narrow fashion possible, creating an ivory tower limited by his skill set and attitude. Good engineers seize a hard problem and drive it to completion, even when that problem pushes them well beyond their comfort zone. The recluse, however, will use any excuse to make something someone else’s problem. Even when the problem falls within his limited domain, he will solve only the smallest portion of the problem, preferring to file a bug to have someone else finish the overall work. When confronted on architectural issues, he will often agree to do it your way, but then does it his way anyway. Months later it can turn out he never understood what you had said in the first place or discussed it in the interim, and by then it’s too late to undo the damage done.

All of us have the potential for these anti-patterns in us. It’s only through regular introspection and frank discussions with colleagues that we can hope to have enough self awareness to avoid going down these paths. Most importantly, we all need to work to create a strong engineering culture where it is impossible for these pathologies to thrive. Once these pathologies become a fixture in a culture, they breed similar mentalities as the organization grows and can be impossible to eradicate at scale.