In October 2005, longtime partner-in-crime Mike Shapiro and I were taking stock. Along with Adam Leventhal, we had just finished DTrace — and Mike had finished up another substantial body of work in FMA — and we were beginning to wonder about what was next. As we looked at Solaris 10, we saw an incredible building block — the best, we felt, ever made, with revolutionary technologies like ZFS, DTrace, FMA, SMF and so on. But we also saw something lacking: despite being a great foundation, the truth was that the technology wasn’t being used in many essential tasks in information infrastructure, from routing packets to storing blocks to making files available over the network. This last one especially grated: despite having invented network attached storage with NFS in 1983, and despite having the necessary components to efficiently serve files built into the system, and despite having exciting hardware like Thumper and despite having absolutely killer technologies like ZFS and DTrace, Sun had no share — none — of the NAS market.
As we reflected on why this was so — why, despite having so many of the necessary parts Sun had not been able to put together a compelling integrated product — we realized that part of the problem was organizational: if we wanted to go solve this problem, it was clear that we could not do it from the confines of a software organization. With this in mind, we requested a meeting with Greg Papadopoulos, Sun’s CTO, to brainstorm. Greg quickly agreed to a meeting, and Mike and I went to his office to chat. We described the problem that we wanted to solve: integrate Sun’s industry-leading components together and build on them to develop a killer NAS box — one with differentiators only made possible by our technology. Greg listened intently as we made our pitch, and then something unexpected happened — something that tells you a lot about Sun: Greg rose from his chair and exclaimed, “let’s do it!” Mike and I were caught a bit flat-footed; we had expected a much safer, more traditional answer — like “let’s commission a task force!” or something — and instead here was Greg jumping out in front: “Get me a presentation that gives some of the detail of what you want to do, and I’ll talk to Jonathan and Scott about it!”
Back in the hallway, Mike and I looked at each other, still somewhat in disbelief that Greg had been not just receptive, but so explicitly encouraging. Mike said to me exactly what I was thinking: “Well, I guess we’re doing this!”
With that, Mike and I pulled into a nearby conference room, and we sat down with a new focus. This was neither academic exercise nor idle chatter over drinks — we now needed to think about what specifically separated our building blocks from a NAS appliance. With that, we started writing missing technologies on the whiteboard, which soon became crowded with things like browser-based management, clustering, e-mail alerts, reports, integrated fault management, seamless upgrades and rollbacks, and so on. When the whiteboard was full and we took a look at all of it, the light went on: virtually none of this stuff was specific to NAS. At that instant, we realized that the NAS problem was but one example of a larger problem, and that the infrastructure to build fully-integrated, special-purpose systems was itself general-purpose across those special purposes!
We had a clear picture of what we wanted to go do. We put our thoughts into a presentation that we entitled “A Problem, An Opportunity & An Idea” (of which I have made available a redacted version) and sent that to Greg. A week or so later, we had a con-call with Greg, in which he gave us the news from Scott and Jonathan: they bought it. It was time to put together a business plan, build our team and get going.
Now Mike and I went into overdrive. First, we needed a name. I don’t know how long he had been thinking about it, or how it struck him, but Mike said that he was thinking of the name “Fishworks”, it not only being a distinct name that paid homage to a storied engineering tradition (and with an oblique Simpsons reference to boot), but one that also embedded an apt acronym: “FISH”, Mike explained, stood for “fully-integrated software and hardware” — which is exactly what we wanted to go build. I agreed that it captured us perfectly — and Fishworks was born.
We built our team — including Adam, Eric and Keith — and on February 15, 2006, we got to work. Over the next two and a half years, we went through many changes: our team grew to include Brendan, Greg, Cindi, Bill, Dave and Todd; our technological vision expanded as we saw the exciting potential of the flash revolution; and our product scope was broadened through hundreds of conversations with potential customers. But through these changes our fundamental vision remained intact: that we would build a general purpose appliance kit — and that we would use it to build a world-beating NAS appliance. Today, at long last, the first harvest from this long labor is available: the Sun Storage 7110, Sun Storage 7210 and Sun Storage 7410.
It is deeply satisfying to see these products come to market, especially because the differentiators that we so boldly predicted to Sun’s executives so long ago have not only come to fruition, they are also delivering on our promise to set the product apart in the marketplace. Of these, I am especially proud of our DTrace-based appliance analytics. With analytics, we sought to harness the great power of DTrace: its ability to answer ad hoc questions that are phrased in terms of the system’s abstractions instead of its implementation. We saw an acute need for this in network storage, where even market-leading products cannot answer the most basic of questions: “what am I serving and to whom?” The key, of course, was to capture the strength of DTrace visually — and the trick was to give up enough of the arbitrary latitude of DTrace to allow for strictly visual interactions without giving up so much as to unnecessarily limit the power of the facility.
I believe that the result — which you can sample in this screenshot — does more than simply strike the balance: we have come up with ways to visualize and interact with data that actually function as a force multiplier for the underlying instrumentation technology. So not only does analytics bring the power of DTrace to a much broader spectrum of technologists, it also — thanks to the wonders of the visual cortex — has much greater utility than just DTrace alone. (Or, as one hardened veteran of command line interfaces put it to me, “this is the one GUI that I actually want to use!”)
There is much to describe about analytics, and for those interested in a reasonably detailed guided tour of the facility, check out this presentation on analytics that I will be giving later this week at Sun’s Customer Engineering Conference in Las Vegas. While the screenshots in that presentation are illustrative, the power of analytics (like DTrace before it) is in actually seeing it for yourself, in real-time. You can get a flavor for that in this video, in which Mike and I demonstrate and discuss analytics. (That video is part of a larger Inside Fishworks series that takes you through many elements of our team and the product.) While the video is great, it still can’t compare to seeing analytics in your own environment — and for that, you should contact your Sun rep or Sun reseller and arrange to test drive an appliance yourself. Or if you’re the impatient show-me-now kind, download this VMware image that contains a full, working Sun Storage 7000 appliance, with 16 (virtual) disks. Configure the virtual appliance, add a few shares, access them via CIFS, WebDAV, NFS, whatever, and bust out some analytics!
31 Responses
hey, Bryan. Congratulations to you and the whole team! 🙂
Congratulations Bryan. I actually work for a competitor, but I think what you guys are doing is really exciting and wish you the best of luck.
-D
As a customer I look forward to more fishworks products!
As an ex-employee of a Sun Partner I’ll have to say appliances are so much easier to sell to SMBs.
I hope there will be more FISHworks stuff coming outside the storage world…
Good work! Congrats!
Sad that we can’t get the write accelerators seperatly or use fishworks with the X4150s.
Are you planning to extend the VMWare image towards the Sun Amazon EC2 cloud product?
It’d be rather neat to be able to say ‘aws run-image i-sunstorage7000’ and be able to try it out just like that.
And when are you doing the same for MySQL? An appliance with this kind of integrated transparency that lets you get to the root of problems so that you can make decisions based on facts instead of guesses?
Bryan and team,
Awesome, frickin, awesome work. The whole FISHworks stack is so cool and extends data storage management and diagnosis skills to the less experienced out there.
My only wish would be for analytics to be bundled in Solaris. That would make every SA’s day!
Make the dream come true and let’s turn this ship around!
Bryan, Mike & team: Bloody excellent work. I only wish I could have been part of the team! Finally someone has put some serious engineering talent into the NAS and storage problem space. I/O, in term of throughput, IOPS and latency is the current challenge in building almost *any* modern, high performance solution. And, before Fishworks the "I/O curve" was as flat as a pancake and being further and further eclipsed by the rapidly increasing CPU "horsepower" and system memory capacity curves (both following Moores’ law very nicely).
With 2Gb DDR2 DIMMs delivering RAM at ~$9/Gb, how about a nice PCIe card based memory cache or even a similar product in a 3 1/2" form factor.
Many congrats Team Fishworks. Thank You for advancing the state-of-the-art….
It’s great to see you surface again Bryan 🙂
Fishworks looks like a real game changer, I look forward to seeing where this takes the storgae market over the next few years…
Looks super sweet!! BTW you’re an awesome public speaker and presenter Bryan 🙂
The VMware image was a stroke of pure genious, not that you guys don’t have that every day.
My sincere congratulations on FISHworks, you beat me to it.
d**n – just a few days too late for us to be taken into consideration, so we’re left with a tool made by a competitor of yours in this field simply because our Sun salesperson couldn’t offer us something like that right in time… 🙁
Great work nevertheless, good luck with this project as it seems just the logical next step in messing up the storage / NAS market.
K.
this is a great story! thank you for sharing it! CONGRATULATIONS to the entire team!!
Hey Brian et al.!
Really good stuff here! A couple of comments:
1. Change the default user name to Administrator, not root. With all respect to
Windows folks, even seeing the word "root" creates angst. Using
Administrator has all the warm and fuzzy hugs they want.
2. To all the folks who want free access to the complete software stack for
their home servers, reconfig the vmware simulator – performance is good
enough for serving up music, video and files to Linux, Windows and Mac
client at our home.
3. The appliance strategy is the way to go – you should build a MySQL appliance
on top of a hybrid storage pool and make it as easy to admin as the 7000.
4. While you’re at it, why not build a suite of middle appliances: email, ldap,
idm, etc. Make them as easy to implement and use and watch your sales grow.
5. Why would you announce this stuff at CEC? Most of the Sun SE’s are at CEC –
they should be out there pounding the pavement with this stuff.
(I will never understand Sun’s marketing! )
Bryan,
Congrats on seeing your vision become a reality! And thx for sharing the story behind the story.
Claire
Awesome, thanks !
This may be out of scope : do you plan to release similar tools for other platforms ?
We have a bunch of X4500/X4540 here that are used for HPC storage (mainly dCache). No appliance, no NFSv4 but we would benefit of such kind of analysis framework on the zpool, cpu and application sides.
congratulations to The Team and looking towards more interesting fisherworks products in future.
Bryan, HUGE fan of DTrace & ZFS. Looking for excuse to use it in our next deployment. However, Sun keeps fumbling and making things hard to get to. Tried downloading the vmware image you pointed to. Filled all the registration, confirmed my email, etc.. Try to download: Download could not be authorized – please contact customer service. Even MORE stuff to fill out. Pass…
Anyway, wish Sun would get out of the way of getting your cool tech in the hands of development groups who might wanna actually do something with it. 😉
Really like the new products — they are a welcome addition into this space. I worry though that one segment that continues to be ignored is a system for small virtualization workloads. Sure, these will be great for that but they (seem to) miss a critical piece in the low end products — high availability. There are a lot of customers (like me) who don’t have the need or the budget for all the whiz-bang features of a system that "starts at" 50K (IE by the time you add enough to make it fault tolerant you spend 100K) but DO really want and have a tremendous need for a little box with 2-5TB or so of hybrid ram/flash/disk storage that is internally fully redundant. I can really only find one product that fits the bill — the EMC AX150 and 150i and even though they don’t fail, they SUCK.
Please give us something in this space for ~25K and it will get eaten up!
Finally, someone at Sun has seen the light on delivering products that could really sell! Doesn’t surprise me that it was you two maniacs, Bryan and Mike…
I completely agree with trickydick. LDAP and MySQL appliances with your analytics built in is a no brainer. Would love to see middleware appliances as well for things like JMS. There is a reason the appliance market is still getting business and it’s because the biggest cost of most operations is people to engineer and run these things. Getting people to engineer this stuff in house is getting expensive and engineering analytics in house is even more costly.
Make sure marketing doesn’t screw this up. Stress the ease of the UI…
Hey Everyone,
First, thanks for the many kind words — the response here has been a bit overwhelming! Apologies for not having been more active on responding to comments; it seems that every second since the launch, we’ve been talking about this product here at CEC. (We’ve had many great conversations, and it’s clear that Sun and — more importantly — our customers are really excited about this new product line.) Some specific responses:
trickdick: you can actually create an "admin" (or "Administrator") account that has whatever authorizations that you would like. I think the larger issue is that we need to have a way for the customizations on an appliance to be easily templated and applied to new appliances. We already allow this to a certain degree with our scripting facility (more on this in a subsequent blog post), but there’s more we can do here — stay tuned.
trickydick and others on appliance ideas: You bet! As you saw from the presentation we gave Greg, this is very much a part of our vision, and your specific ideas are welcomed! While I hesitate to commit to anything, the MySQL appliance and the LDAP appliance are two examples that have come up a LOT — so stay tuned here, as well.
Ben: sorry to hear about your experience! I’ll be e-mailing you directly to get this resolved.
John: yes, we’ve heard this quite a bit! We have some specific plans in this area; expect a direct e-mail from me, as we would like to talk with you about some of the compromises involved to get an HA solution in that price band, and better understand your needs. But I think you’re going to like what we’re thinking!
Bryan, will this AWESOME stuff (analytics, etc) be available as a separate product?
Well done, guys.
"Make sure marketing doesn’t screw this up."
Please, don’t sleep on laurel. There is long way to go.
Just some ideas:
– Separate appliance SW and sell it to partners, OEMS(even HP, IBM, Dell or even Apple), but force their customers to buy support from Sun directly and keep customers warning that it is optimized for SUN HW. Let competitors do work for you. Spread it into HP Proliants and IBM x-series installed base. Convert old HP’s, IBM’s and Dell’s to Sun install base.
I guess it is easy as OpenSolaris runs perfect on them.
– Make NAS gateway for traditional storage and JBODs
– Add Analytics BUI to Solaris. I bet that Solaris admins would pay for it or distribute it with Solaris gold and up support. Dtrace needs it badly.
– KIS (keep it simple)
Thumbs up.
Great work! I’ve spent much of the weekend playing around with the vmware image (definitely the right move to get the software out FAST). I too would love to see a SME-scale device with an HA option. I’m currently using netapp, but need to upgrade it because the darn thing is terribly CPU bound, and frankly I’d much prefer to go with a fishworks appliance if I can make it fit the bill.
I need more storage than the 7110 though, and the 7210 is really overkill, so I’m sorta stuck in the middle like John. HA support would be GREAT too. Do contact me if you’re looking for use cases or want to talk tradeoffs.
Also, like the guy above said, don’t let marketing screw this up. Secondly, don’t let your supply chain screw it up! Last time I ordered a system from sun it took 3 months to get to me, and they couldn’t even tell me where it was until a semi-truck showed up at my office to drop off my single x4100. Heck, I’d probably try-and-buy one now but I’m not sure I have the energy to hassle with it!
Congrats Bry! So excited to see this project out in the public. I know you guys worked your ass off to get this out the door, so I’m excited to see it come together as such an amazing project. It is also really refreshing to see the transparency and visibility you gave everybody into the process.
(I’m embarrassed to say the first time I submitted this comment I failed the math question. I blame our schools.)
Bryan, I not only work for a competitor you are going after but I also
worked in Solaris in a previous life. Somehow I guess I bleed purple!
Lets just say that the SEs here are running scared as in head-to-head
deal after deal, you guys are coming in at half our discounted price!
The only ‘defense’ against this project and the 7000 series machines is
to try to point potential customers to Sun’s stock price, its prospects,
FUD about Sun being bought over, no reference customers, etc. In other
words, Fishworks seems to be impossible to defend on merit. I talk to
our larger accounts all the time and let me tell you- if Sun puts some
smart and aggressive sales guys on this- many of your competitors-
including us- are toast! Please understand though that you do
need to pay attention to execution not just innovation.
Anon,
Interesting comment! And yes, the counterattack — such as there has been one at all — has been what we had expected: that Sun has a terrible track record in storage (true, but not relevant given the history of this project), that Sun’s stock price is in the tank (certainly true, but so is everyone else’s), that Sun is going out of business (not true — we’re throwing off too much positive cash to be going out of business any time soon), and that there are no reference customers (true, but only because many of our customers are also customers of competitors’ products, and don’t want to needlessly antagonize their suppliers by going on the record). As you say, none of these attacks are on the merits of the technology — in part because the three advantages of our product (disruptive price/performance, hybrid storage pools, analytics) are quite difficult to attack without a product in hand that doesn’t at least match our offering. And your point about the need to execute is also well taken. Fortunately, many at Sun recognize this, and there is much attention being paid to execution in this product line. Finally, I can corroborate what you’re seeing: we’re having many, many conversations with many competitors’ customers — with virtually everyone incredibly excited about the prospects of the 7000 series. Should make for interesting times!
Bryan,
Congrats on the fine product. Looking through your proposal PDF…what about this Sunray Appliance idea ??? Any chance that his will get any consideration ? A Sunray server that doesn’t require manual care and feeding would enable the technology to go places where it currently can’t be practical (specifically, where there are no Unix admins). Keep up the good work.
So is there a chance that the core fishworks appliance technology
would ever be available for individual developers to
use as a tool to assist building Solaris appliances with?
Right now, the only decent tool i’m aware of for building appliances
is rBuilder, which is of course Linux-specific.
I would be very interested to develop appliances that would run well on
Solaris and Sun servers; it seems like a terrible waste to use such stout
hardware and install such software as mainline Linux kernels that lacks any
form of real fault management.
As I think Solaris is more stable and suited for things like a simple router/web filter appliance, or MySQL / web server appliance than Linux is.
But there’s very little guidance I could ever find on how to do it and
have a reliable thing that would not sometimes require administration
by a Solaris guru.
Excellent innovative work. Congratulations to you and your entire team.