FAQ: STAC Benchmark Council (rev 1.8)

What is the STAC Benchmark Council?

The STAC Benchmark Council is a vendor-neutral body facilitated by the Securities Technology Analysis Center (STAC?) whose purpose is to develop and maintain standard benchmark specifications for technology used in the capital markets. It is also a forum to gain in-depth knowledge of new technologies and benchmarking best practices. The council consists of trading firms ("customers") and providers of technology used in the trading process ("vendors"). Vendors provide input to the standards, but only customers have voting rights.

What problems is the Council designed to solve?

The purpose of the Council is to create benchmark specifications that make it easier for both customers and vendors to understand the performance of different securities technologies. Trading firms today face increasingly difficult technical requirements, as competition and regulation force them to reduce latency and cope with increasing transaction and data volumes. Vendors are offering major performance innovations at every layer of the technology "stack"—from more efficient software to more advanced processors and faster networks. But evaluating this new technology accurately and quickly has become a key challenge for the industry. On an ongoing basis, customers need answers to questions such as:

- “Is my system as fast as the technology available, or am I at a disadvantage?”

- “Can I speed up my system by changing one or more parts of my stack? What are the truly important changes that I should evaluate?”

- “How much equipment will I need to keep up with volumes next year?”

Most customers simply don’t have the resources to test the constant stream of new software platforms, chips, accelerator boards, servers, operating systems, networking technologies, storage mechanisms, and so on. Vendors sometimes publish benchmarks, but the methodologies and measurements are not standardized. This makes it difficult for customers to make apples-to-apples comparisons between new technologies and their existing systems. Sometimes benchmarks lack detail and don’t measure a lot of the things customers care about. This lack of transparency also reduces innovation. If a particular vendor doesn’t have hard data regarding areas where its performance falls behind competitors, it has less incentive to improve those aspects of its performance. These problems are a classic prescription for benchmarking standards.

What are STAC Benchmarks?

STAC Benchmarks measure the performance of key capital markets workloads when applied to a given "stack under test." The benchmark requirements currently identified fall into three sets, corresponding to the three basic steps in making a trade:

- STAC-M (market data): benchmarks based on workloads such as direct exchange-feed integration, market data distribution, tick storage and retrieval, etc.

- STAC-A (analysis): benchmarks based on workloads such as trading algorithms, price generation, risk calculation, etc.

- STAC-E (execution): benchmarks based on workloads such as smart order routing, execution-related communication, etc.

For details on how these break down into sub-domains, see www.STACresearch.com/domains.

STAC Benchmarks are oriented around workloads (inputs and outputs) rather than particular categories of technology, because technologies change quite quickly. However, they will measure the capabilities of software such as market data systems, messaging middleware, in-memory caching, and complex event processing systems (CEP). They will also quantify the impact of new underlying technologies. Today those include things like hardware-based feed and messaging solutions, hardware-based analytics accelerators, compute and data grid solutions, InfiniBand and 10-gigabit Ethernet networks, multicore processors, and the latest operating system and server technologies. A fundamental goal of STAC Benchmarks is to be meaningful to technology buyers.

Most of the benchmarks aim to simulate real-world conditions (or future real world conditions) as closely as possible. Some of them may examine boundary conditions that reveal a stack’s performance tradeoffs, but only if understanding those tradeoffs is important to customers. STAC Benchmarks are not constructed to create big numbers for the sake of publicity. While STAC Benchmarks are about performance, they also put that performance into a business context by relating it to key cost drivers such as power, space, and other operational resources. For example, the STAC-M1.OPRA specs include metrics such as "messages per second per watt," and current Council members tell us those kinds of metrics will continue to be crucial.

How does the Council make decisions?

Both customers and vendors have a crucial role to play. Customers have decision authority (voting rights) over the benchmark specifications. Large firms (1000 or more employees) have three votes, and small firms (less than 1000 employees) have one vote. Vendors may make proposals and commentary but do not have voting rights. Vendors will lobby for standards that highlight the benefits of their products. This is a healthy process that will ensure that the standards are as comprehensive as possible. Reserving voting rights for customers will ensure that the benchmarks focus on the business needs that are most important to the market.

What do customers get with their membership, and what are the benefits?

The Council is essentially the “inside track” at STAC. As Voting Members, customers get the following:

- Opportunity to influence the benchmark standards

- Opportunity to vote on the standards

- Briefing sessions and test documentation that go into more depth than what is made available to the public

As the Council grows and we collect more feedback from Members, this list will probably grow to include other services that aid in your own technology discovery and testing processes. The benefits you may gain through participation include:

- Intelligence. Faster and deeper insights into the strengths and weaknesses of new technologies and optimization techniques, relative to each other and relative to those you use today.

- Best practices. Improving your own testing competencies by picking up benchmarking techniques.

- Better products. Benchmark standards give vendors more insight into how their performance compares to their competitors and motivates them to improve their own performance. This obviously benefits customers.

- Perspective. Insight into what’s going on in different areas of the trading process. In large organizations such as a global trading firm, there is usually a high degree of specialization. Each employee has a detailed view into one trading-technology silo (e.g., market data, FIX messaging, foreign exchange pricing engines, and so on). The STAC Benchmark Council is a way to find out what’s going on in other areas of the trading process. This may present opportunities to cross-fertilize best practices or to advance your career by developing a deeper understanding of the issues and technologies in domains beyond those you’re concerned with day-to-day.

What do vendors get with their membership, and what are the benefits?

As Contributing Members of the Council, vendors get the following:

- Opportunities to observe and participate in key aspects of the specification process

- Opportunities to submit additions and modifications for consideration by the Council’s Voting Members

- Access to detailed information from benchmark projects

Again, this list is expected to grow as we gain feedback from Contributing Members. The benefits that come along with these include:

- Influence. Contributing Members have the chance to ensure that the benchmark specifications provide an avenue to demonstrate their products’ strengths.

- Brand leverage. Participation in an effort to establish objective measurement standards is a statement that a vendor is confident in its performance and dedicated to improving its performance.

- Credibility. An objective benchmark executed according to agreed industry standards is one of the best advertisements a vendor can hope for.

- Optional outreach opportunities. Vendors will be able to sponsor briefing sessions (webinars, breakfasts, other events) for customer members in which the vendor can promote its products, gather feedback from customers, and establish sales contact with those customers who express interest in follow-up.

- Intelligence. Once benchmarks are standardized, vendors will gain better insight into how their performance compares to that of their competitors. And like customers, they can incorporate benchmarking best practices into their own testing regimes.

Can other kinds of organizations join the Council?

The Council also welcomes the participation of other organizations involved in trading, especially exchanges. Exchanges are large consumers of trading technology. Whether the benchmarks standards chosen by trading firms will also reflect the priorities and requirements of exchanges is currently an open question, so it is not clear whether a separate set of standards is required. (Initiatives such as MiFID that blur the boundaries between trading firms and exchanges may ultimately drive convergence no matter what.)

At the moment, membership is not open to other types of institutions. We are considering the appropriate ways to involve organizations such as universities, the press, and consultancies.

Is the Council global in scope?

Absolutely. The Council welcomes customers and vendors from all geographic regions. Its charter members are some of the largest banks who trade in every market around the globe. The Council representatives from these firms sit on both sides of the Atlantic. While the bulk of the Council’s communication is electronic (hence borderless), there are also in-person meetings for both New York and London. Plans are also forming for meetings in other European and Asian trading centers.

If I join the council, what is expected of me?

That is up to you. If you are with a customer firm, your firm will have a vote. We would encourage you to get involved in the issues you care about and contact your firm’s vote holder. There are discussion forums within the Council’s online community, teleconferences, and periodic in-person meetings. Participation in all of these is completely voluntary.

What is the Council working on right now?

The Council's initial focus was on STAC-M1 Benchmarks (direct feed integration), the first version of which was approved on 22 September 2008. The STAC-A1 working group is now hard at work specifying event processing benchmarks. Specifications have been drafted for both STAC-M2 (market data distribution) and STAC-E2 (transactional communication), which will be useful for benchmarking messaging systems. See www.STACresearch.com/domains for the complete overview of workload domains currently in scope. For more information on what it means to join a working group, see www.STACresearch.com/workinggroups.

What does it cost to join?

The discounted fees for the Council’s inaugural year have been extended for an additional month, through October 15, 2008. For customers with 1000 or more employees, this fee is USD 5,000. For customers with fewer than 1000 employees, it is USD 2,500 and for vendors it is USD 5,000, irrespective of firm size.

How do I join the STAC Benchmark Council?

That’s simple. Send an email to council@STACresearch.com. We’ll check whether your company is already a member. If so, we will let you know who your firm’s primary representative is. If not, we’ll contact you to sign up.