Explore our Blogs

Over the past few months I got my first chance to work with Kafka Streams. For those who don’t know, Kafka Streams is a new feature in Kafka 0.10.0 that adds stream processing capabilities into the existing Kafka platform.

Josh Perryman visited the Northwest in July 2016 for DataDay Seattle and gave an updated version of my Graph Database Shootout.

Continuous Discussions is a weekly online panel discussion which is sponsored by Electric Cloud and discusses a variety of topics around Agile, Continuous Delivery and DevOps with a range of different speakers.

How can you retool your SQL talent to deploy on your graph project? According to Google Trends, interest in graph databases is increasing while SQL interest is level or even decreasing – but SQL searches are still 43 times more prevalent than graph.

Achieving user adoption goals for enterprise software can be challenging.

The Austin Data Geeks & Austin Graphs Meet-ups invited Josh Perryman to do an encore presentation of our GraphDay presentation: Graph Database Engine Shoot-out.

A case study at Graph Day recounting a client study we did to see whether their database could be reorganized to offer improved query performance. We looked at graph databases (OrientDB, Titan, Neo4J) because they thought of their data as graph data, and relational (Postgresql) because that’s what their database already implemented.

‍We are in an era of unprecedented innovation in databases. Data-intensive companies are grappling with whether the many new options — NoSQL, Key-Value, Document, Column Family, Column-Oriented — are appropriate for them. The commercial success of Facebook and LinkedIn makes graph databases a hot area of investigation. Unlike many new databases, they are not a variation on or a simplification of relational databases. Instead they require new ways of thinking and modeling data. In return they can answer truly novel questions.

If your application does anything interesting, it has secrets. No, not that no one remembers how your code works, or that that one dev put a curse word in his checkin once. Not those kind of secrets. We’re talking about database passwords, secret API keys, per-user encryption keys, and the like. For a long time, web developers put those in configuration files or read them from files which programmers promised never to accidentally check in. Cross your heart.

In this post we are going to explore a slightly more complicated yet common scenario involving continuous deployment of a .NET web application.

We are meeting more people who are interested in looking into the world of graph databases. Palladium has executed proofs–of–concept for clients to help them explore this world. In this post we summarize what sorts of questions we feel like a proof of concept project can answer, and how we typically tackle them. For our presentation at Graph Day, we’ll be walking through one in particular, but really there are a variety of answers you may want.

In this post we summarize what sorts of questions we feel like a proof of concept project around graph databases can answer, and how we typically tackle them.

In this post we will examine a simple scenario involving the deployment of a .NET based web application to an Azure Web App using Azure Key Vault for secrets management. All the code used for this blog post can be found here.

Everyone gets delusions of grandeur!” That’s what Han Solo said after being frozen in carbonite. I’ve been solving data problems for customers the last year and a half and am now getting back in graph DBMSs. We took a nice look at Titan last week, can’t wait to play with that some more. I’m going to give a bit of the same to Neo4j. All of this as prep for my talk at GraphDay 2016 in Austin, TX.

Smartphone and tablet experiences are a major force influencing product development, irrespective of industry, according to an internal survey we recently conducted with our clients. Business users want to access their software applications from a myriad of devices, all behaving correctly regardless of form factor.

As part of the work we’re doing to refresh our graph database evaluation for a couple of clients (and our upcoming talk at Graph Day!) we took Titan 1.0out for a spin last week. We’ll be doing more in-depth explorations on some in-house and public datasets over the next few weeks, but here’s some preliminary impressions based on a contrast with the Titan we came to know a year ago or so.

Our client’s legacy system held graph-like data in a relational database, but new customers’ data sizes were crippling performance and scale. As part of an overall architectural rejuvenation, we evaluated migrating their data to graph and relational schemas to determine if query performance and scalability could be improved. With representative data in hand, we designed alternate relational schemas, graph database designs, and triple store designs, benchmarking performance and noting subjective measures such as ease of use and fluency of the query language. Vendors included PostgreSQL, Neo4J, Titan, and AllegroGraph. Follow-up studies included other vendors. The results surprised us, leading to a hybrid relational and graph recommendation. We have implemented the first milestone over the last year. Follow-up work shows that graph DB vendors have come a long way even in that time. This methodology and information in this case study should be useful to teams choosing a database engine, whether graph or relational, for their next project.Our client’s legacy system held graph-like data in a relational database, but new customers’ data sizes were crippling performance and scale. As part of an overall architectural rejuvenation, we evaluated migrating their data to graph and relational schemas to determine if query performance and scalability could be improved. With representative data in hand, we designed alternate relational schemas, graph database designs, and triple store designs, benchmarking performance and noting subjective measures such as ease of use and fluency of the query language. Vendors included PostgreSQL, Neo4J, Titan, and AllegroGraph. Follow-up studies included other vendors. The results surprised us, leading to a hybrid relational and graph recommendation. We have implemented the first milestone over the last year. Follow-up work shows that graph DB vendors have come a long way even in that time. This methodology and information in this case study should be useful to teams choosing a database engine, whether graph or relational, for their next project.

Unfortunately, I had to miss the live Apple event last week. So I took a little time this weekend to wade through it and came across this rather magical little multi-touch moment from the iPad Pro demonstration. To the untrained eye, simultaneously using a virtual ruler while drawing a line with a stylus may seem like standard fare. As a practitioner who has created multi-touch user experiences involving both stylus and touch inputs, I realize that this moment Apple depicts is not to be taken for granted. Here’s a breakdown of what’s likely happening to make this moment possible:

At Expero, we don’t like to just guess. We do under certain time constraints, but we want to get it right. That means finding out what the right design should be—and that means thoroughly evaluating options, often with feedback from stakeholders and end-users.

What do my users want to do with Big Data? How do they want to visualize it, interact with it and manage it? How big is my data, really? How much data can a human deal with at one time, and how much data should we process at one time? How can the UX accommodate data sources that respond at different rates?

If you want to store lots of data, the Amazon cloud has buckets and glaciers for you, but not a shared file system.

asdlkf

Are you a Product Manager looking to show some early design directions to stakeholders and end-users? Are you a Developer needing a little more direction from the design team? Or a UX Designer or Researcher trying to figure out how detailed your designs need to be to communicate your intentions and get valuable feedback from users?

This post describes how to debug some library dependency issues on a Linux machine. I built a nightly version of Julia (a language for technical computing that we’re pretty excited about here), on Linux, deployed it to a different machine, but then it failed to launch, complaining about

As a user researcher, I am, of course, an evangelist of all things research. I love research! I love reading everything and learning everything and understanding everything I possibly can about a subject. Infographics are neat but I want more more more information! Give it to me—all of it! As a user researcher, I also, of course, realize that not everybody feels the same way. Often including clients.

I recently finished cleaning up our build system and adding support for both client-side and server-side unit tests. The build is cross-platform, meaning our developers can build and run the application whether they are on Windows, OSX, or Linux.

Ever wonder about the best way to visualize your application’s data sets?

A design pattern, as we explained in our January 2015 blog post, is “a good, reusable solution for a common but specific interaction design problem.” In this online seminar, Lynn Pausic and I present design problems that are often seen in applications and sites with large or complex data sets. We also give 10 examples of patterns that can be used to solve those problems–things like how to set up a complex data table to view status at a glance, or when to employ column filtering so users see only what they choose to see.

What if, no matter how you try to simplify, your aggregate root is pretty darn big? Writing application services to handle these large entities is a challenge. We run into this all the time with scientific computing.

I’m a fan of Yahoo for news and email, but I’m not a fan of their new “two-tap rule.” The rule states that once a user is in a Yahoo application, it should take only two taps to do anything the user wants to do. Fast-to-use apps are the goal here. The two-tap rule sounds a lot like our field’s misguided “three-click rule,” which Expero cautioned against in 2007.

Several years ago, I attended a neat speech by a young but awesome UX Designer named Golden Krishna, with credentials ranging from product innovating at Samsung, to designing at Zappos, to writing for major publications. The speech was titled “The Best Interface Is No Interface,” and the room was packed—presumably full of user interface designers. It was a great hook.

Is Google’s Material Design for complex software? My first reaction after reading through these new Google guidelines was: no. There’s no way these simple rules will translate seamlessly to complex enterprise-class applications—the kinds of projects that Expero excels at—at least not without starting over and removing 90% of the features/functionality that users want at their fingertips.

Fascinating though it is, I’m happy to observe prison life from the outside through shows like Oz or Orange is the New Black. It’s the strange way prison mirrors the outside world that’s so compelling. They have police (gangs) and wars (gangs) and commerce (smuggling) and currency (cigarettes, stamps, etc.) just the same as the free world.

Many organizations haven’t embraced UI design patterns, despite their growing popularity among UX professionals. Those organizations are missing out on a ton of benefits for Product Management, Engineering, Marketing, UX and the business as a whole.

It’s hard to hire good developers. We face the same struggles everyone does in sorting the good from the bad. One of my favorite tropes is the resume as failed inductive proof.

In part 1 of this series, we looked at how using an IOC container helped us separate the concerns of the construction of a ZookeeperClient from the concerns of using it in service handlers. In this one, we look at how the IOC container can transparently help us manage a singleton which leaks memory.

IOC containers elicit strong emotions from programmers, ranging from reverence to disdain.

In the second part of this blog post we are going to look at a simple methodology for applications to take advantage of a centralized and transparent configuration store. I had thought this was going to be the hardest part of this project yet it turned out to be the most straightforward. This was done by implementing our own ProtectedConfigurationProvider class which gave us the ability to inject configuration data at runtime from any source available to me. I settled on this approach based on a lot of searching but which has been nicely condensed in several blog posts you can read here and here.

Our product suite (all .NET based) consists of a small but growing number of related applications with a lot of configuration data stored in app/web.config files with multiple environmental transforms. Much of the configuration is repeated across multiple projects in each environment (e.g. connection strings, email servers, etc.) so centralization of this data would be of great help. Currently, making even simple changes to any of these keys requires sifting through a large number of config files and transforms to make changes which is tedious, error prone, and difficult to test effectively.

ISPC Performance

Julia Performance

This the third post in a series, which started with a description of the problem, and continued with an F# benchmark.

There is still a substantial gap between this result and the result we’ll find with other environments, and my guess is this is a code generation issue, i.e. instruction selection and scheduling, but I’m not an expert in this area either!

I’ve got 405MB of 3D seismic data from Teapot Dome sitting in my file cache, and I want to give you a quick view of some of its summary statistics. How long do you think you should have to wait? If you’re working in Excel, you might be happy with a few minutes. A .NET programmer — used to endless database calls and virtual machines in his line of work — wouldn’t be too surprised at a few seconds, or tens of seconds. Long enough to fire up a spinny cursor and send you to Facebook, or whatever your work-day sin is.

In a previous post, we talked about untangling multiple UI controls so that they could be developed independently, but react to user interaction in a synchronized manner. Let’s posit for a moment updates to a line on one map control should cause re-rendering of a cross-section control associated with that line, but that the two controls are in different browsers, or even on different machines.

Reactive Synchronized UI Components: how I learned to stop worrying and love synchronized maps and cross-sections.

Recently I’ve come across the need to use an ambient logical context that would automatically flow with my program logic as it jumps from thread to thread. The .NET Framework has support for this when you are working with Threads and Tasks. It is not well documented, but it Just Works…mostly.

One of ZooKeeper’s nice features is the ability to set up a watch on a node, and be updated whenever it changes.

ZooKeeper’s “native” client APIs are C and Java. If you’re programming in .NET (or Python, or a few other languages), the docs helpfully point out that some friendlies have programmed clients that “might” work for you. “Might” is frustrating, as is the possibility that the libraries are behind. So we used the Java version anyway, and made it a little more idiomatic .NET. It turns out to be a nice look at how to use Java from .NET, and how to implement Task and IObservable patterns by by hand.

Okay, so you’re sold on using ZooKeeper to be your service locator or configuration repository. Your services will all talk to ZooKeeper when they start up to find out who they are, who their neighbors are, and generally how to get on with all the other animals at the zoo. But what service locator do you use to find out where ZooKeeper itself is? (ZooKeeper is actually in one or more places, since it typically runs on multiple servers in a production environment.) The answer probably depends on the scope of your problem: a 10,000 node cluster will be different than a few dozen services. Your best options are drawn from the service locator patterns already built into your OS or environment. Here we’ll talk about 3 options.

In the last post, we used ZooKeeper as a service registry. When services started, they registered with ZooKeeper at a pre-agreed place. (/services/{dataset-name}). Clients could list the data servers available and decide which ones to connect to, or request that new ones could be launched. Thanks to ephemeral nodes, servers can crash and their registry entries are automatically deleted. Today we’ll talk about three use cases for watching changes in ZooKeeper.

Zookeeper is a distributed database originally developed as part of the Hadoop project. It’s spawned several imitators: Consul, etcd, and Doozerd (itself a clone of chubby.) A lot of the material out there about Zookeeper describes just how it works, not necessarily what you’d use it for. In this series of posts, we’ll cover how we used it at one client — and it how it also got abused.

A good development team will already have a battery of tests in place to ensure their code is correct and can be refactored safely.

Turning from the open-source Titan to the commercial AllegroGraph was like stepping out of my 1998 4Runner and taking a spin my boss’s BMW. It was fast. It was sleek. It had all of the modern thingamajiggies that come with new, well-engineered solutions built by companies with the resources to do it well.

It starts innocently enough. You need a database connection string and to know which tables are safe to cache, and there’s just no sense in putting that in your source code. Right? I mean, why put hard-coded stuff in your programming language?

In this third post we look at how the application’s graph data fit into another property graph database: Titan.

Especially early in product development, small custom-built datasets are the order of the day.

In this second post we look at how the application’s graph data fit into a leading graph database: Neo4j. Quite well actually.

Most development is feature driven. A developer is on the line to complete a user story or functional requirement, and even if the application gets a little slower, she’d rather have a demo to show during sprint review, instead of watching every one else’s demo.

Every distributed system eventually requires messages to be written on the wire to be transmitted from one machine to another. In many cases these messages are hidden magic. Using WCF web services, or Thrift RPC, code-generated proxies make remote calls look like function calls.

We talk about data, and how several data concepts such as “Big Data” and “NoSQL” are currently in the vogue. But just as all politics is local, all data is ultimately specific to its own subject domain. Data is not all the same, and so we shouldn’t expect that the general data tools will be the best tools when working with any particular set of data. Choose the right tools with the best fit for your data and you’ll spend more time in analysis and realizing the value of your data, and less time working around the restrictions of your tools.

How many times has a customer come and told you your product was “slow”? In this multi-part series, we will discuss how “slow” happens, and how you can fix it.

We were working with a potential client a few weeks ago, trying to figure out if we could help them improve some seismic processing software. The software had excellent science under the covers, but the visual interface was old and tired. Could Palladium help rejuvenate their user experience? Old looking software can imply old or out of date capabilities. Could we make it, well, better?

Implementing this new logic “the declarative way” is an interesting and typical example of how we use Knockout and Rx to capture the logic and business rules in declarative statements that just work and avoid the complex logic you’d need to implement this functionality imperatively.

Unit of measure conversions are a constant concern in scientific code. Most well written scientific domain kernels should be unit un-aware because the equations of nature are generally unit invariant: momentum is mass times velocity whether velocity is in meters per second or furlongs per fortnight. But there are always important places where the actual values matter: water boils at 100 degrees Celsius. Therefore one typically assumes a set of canonical units in the computational domain to make the programming more straightforward. It’s also more efficient and numerically stable to only translate units on the boundaries of the computation domain, rather than littering them throughout.

Recently I was working on a GUI that walked the user through a series of 5 steps in a Workflow. The user would spend most of their time on the 3rd step. As the user made edits, those edits would immediately be sent to the server for processing, and results from the server would show up in a different part of the GUI. There was a requirement that whenever the user left the 3rd step, we needed to send a message to the server to commit the changes they’d made.

When working with RxJs and Knockout, there will be times when you have an Rx Observable and need a Knockout observable so you can bind it to a view. And there will be times when you have a Knockout observable and need an Rx Observable so you can merge several event sources or some other type of logic that is more easily solved by the RxJs linq operators.

When writing a complex single page web application, you find yourself dealing with A LOT of asynchronous activities. The user is busy clicking on UI elements and typing in text boxes. Asynchronous requests and responses are flitting back and forth between the server and client. Perhaps you’ve even got asynchronous messages coming from web workers.

Up until recently, Firefox was the only Windows 7 browser which had support for multitouch. In a recent build, Chrome has also added experimental support. Unsurprisingly, they’ve used the same touch model used by mobile safari.

Q: I have a new job as a UI designer, and my company just doesn’t understand User Experience. Development runs the show, and they think my job is to design icons and color palettes. What’s the best way to explain UX so that everybody gets it?

Producing great user experiences in an Agile environment can be a challenge. As a designer who has been working in the User Experience (UX) field for over 20 years and as part of Agile teams since 2006, I have truly come to embrace Agile. Dare I say, often I prefer it. As the product releases move ever faster, Agile, Kanban and other Lean methods are here to stay. It’s time for the UX field to get better educated on these topics and embrace change. Many aspects of Agile and Lean methodologies are good for UX. Here are a few tenets of going Agile-Lean that we should be using to our advantage:

There’s an old standby which tells us that A supercomputer is a device for turning compute-bound problems into I/O-bound problems.

Today I had the lovely experience of being told “the network to the cluster is down” while I was writing some code that was supposed to use the cluster. Was I stalled? How could I test my logic? It turns out we’re rather obsessive about separating interface from implementation, usually via C# interface definitions. In this case, I just went down the road I was going down anyway: making some simple mock objects to model the cluster dependencies. (We use Moq.) Now I don’t really care that the network is down.

In the kind of programming we do — scientific simulations and decision support — modeling is usually the first task, and often the hardest. Structuring your problem the right way can make all the difference in determining whether future code is graceful or spaghetti-like.

I had an interesting discussion with a coworker while planning a new feature for our current project. We are practicing Domain Driven Design and maintain a strict separation between the GUI and the domain logic. We actually treat them as separate applications.

Back in the 1990s, if you wanted to interview a C++ programmer, you’d ask him to write a string class. My programming homework counted words or wrote versions of grep(1). Perl made one form of regular expressions popular with the masses.That’s what you cut your teeth on. Now most every language you learn to program in has a Unicode-compliant standard string library which most people don’t think much about anymore. For most of us, it’s a solved problem. (Though there are fun exceptions.)

Q: I’m designing the UI for an application, and people here have different ideas about what this product should do. How am I supposed to design when I don’t know what I’m designing?

Q: How can we find people to be in our studies?

We’re redesigning our site, but there’s internal debate about whether the new design is any better than the current site. What’s the best way to determine which one is better?

Q: We run think-aloud usability tests at my company, but some people here don’t pay attention to the results because we don’t get much “hard” data. Do we need to run different tests?

Q: Someone on our website redesign team wants to include a list of 100+ links at the bottom of every page (to mirror Salesforce.com’s approach). Are a lot of companies including essentially their site map on every page?

Q: How can we optimize our site content for search engines but make sure it’s usable for people, too?

Q: We get feedback from users when we’re able to do usability tests, which is not often enough. What other sources are good for user data?

Q: What are the pros and cons of remote usability testing?

My company’s site has links that say “click here” or “click here for more info”. I think our links should be more specific, but other people here say it’s OK because they see it on other websites. What do you think?

Q: My company thinks about customer requirements but does not think about end-users. What’s your advice?

Q: My company just switched to an Agile Development Process. How can I incorporate user-centered design and research activities into Agile Development?

Q: For setup pages which are generally only used for getting started, do you recommend wizards to guide the user through the steps?

Q: When is a hierarchy too deep, and what are the alternatives to using a deep hierarchy to access and organize information?

Q: Is it important that users be able to get to any content in a website within 3 clicks?

Q: How do you do research for an online wellness program that spans the course of several weeks and measures changes to behavior over time?

Q: How do you design a transactional page that requires horizontal scrolling? Do you freeze the left frame like Excel, or use tabs?

Q: Are there any usability issues with using drag-and-drop controls in a web application?

Q: Which is better – long scrolling pages, or short pages that don’t scroll?

Q: Is it OK to change the navigation of a website once users have logged in?

Q: What are the issues with disabling a browser’s navigation bar?

Q: I am looking for some clarification on best practice for “back to top” links: When should they be used and how? Jakob Nielsen is against them, but this is not a universally held view. They can be very useful for FAQs. What do you think?

Q: Are there any usability guidelines with respect to hyperlinks? In particular, the number of them on any given screen (for example, a page with 50 hyperlinks).

Q: I am the only Information Architect + Designer in an IT Solutions company. I am also fresh out of college. How can I introduce some processes to work with the programmers / coders who are working on Enterprise Solutions?

Q: When evaluating a web site, what is the correct process to create tasks for a user test?