Blog Category: io


Sclable Domain Designer 2.01

By Roland Rust,

Prior to the 2.0 release of Sclable’s Business Application Development Platform, the Domain Designer was part of the generic application and vice versa. As of Platform 2.0 the Domain Designer is not only a standalone application, but also a good showcase of what can be achieved upon Sclable besides extending the generic application.

Extensibility through packages

Sclable’s Domain Designer is a complete re-write and re-design. It is available as a package from the Sclable Factory, our centralized package hub, and can be installed via BOB, our build tool. The Domain Designer allows you to access any of your local or remote Sclable installations via a web service, which is a package of its own. Both packages serve as a good example of how functionality can be packetized when extending Sclable.

Model, visualize and explain complexity

Sclable’s Domain Designer is developed with large and complex business domains in mind. On the one hand we want the tool to let you be quicker than copy and paste. On the other hand it should be as useful for you as a developer when you need to visualize and explain the complexity of a business domain to others.

In-Graph Editing

Since ERD and Workflow graphs are used a lot when modelling business domains, we decided to drop server side rendering with GraphViz in favour of client side rendering with dagre/dagreD3, so you can watch the graphs morph while modifying the domain model. This helps a lot to stay focused, especially when you are editing large graphs, giving the tool a generally more responsive feeling.

Sclable workflow graph. This image shows a workflow from our sample package featuring a ticketing process. The example shows a highlighted action named “set to solved” which is a workflow action of the type “edit”. It lets you set tickets to the state “solved” regardless of their pre-state. This is especially useful for interactive data cleaning after deleting a state when instances of the entity referencing the state get decoupled. Image: "Sclable.com"

Sclable workflow graph. This image shows a workflow from our sample package featuring a ticketing process. The example shows a highlighted action named “set to solved” which is a workflow action of the type “edit”. It lets you set tickets to the state “solved” regardless of their pre-state. This is especially useful for interactive data cleaning after deleting a state when instances of the entity referencing the state get decoupled.
Image: “Sclable.com”

Workflow Edit Action

Actions of type “edit” can have a post-state while not having a pre-state. This allows instances to be set into a specific state from any state. An example use case for this feature is a “cancelled” state within a order workflow for example, when you want an order to be able to be cancelled anytime.

Editing Action Attributes. This image shows how action attributes can be defined as of Sclable 2.0. From ‘invisible’ over ‘read only’, ‘editable’ and ‘required’ different levels of user access to an attribute’s value can be set. What’s happening behind that is quite a complex combination of modelling actions: setting ‘invisible’ in fact deletes an action attribute from the sclable core domain, ‘visible’ adds the action attribute with a boolean flag for ‘readonly’ set to true, ‘editable’ sets the ‘readonly’ boolen to false and ‘required’ creates a corresponding restriction along with the action attribute. Image: "Sclable.com"

Editing Action Attributes. This image shows how action attributes can be defined as of Sclable 2.0. From ‘invisible’ over ‘read only’, ‘editable’ and ‘required’ different levels of user access to an attribute’s value can be set. What’s happening behind that is quite a complex combination of modelling actions: setting ‘invisible’ in fact deletes an action attribute from the sclable core domain, ‘visible’ adds the action attribute with a boolean flag for ‘readonly’ set to true, ‘editable’ sets the ‘readonly’ boolen to false and ‘required’ creates a corresponding restriction along with the action attribute.
Image: “Sclable.com”

Workflow Action Attributes

Attributes can be assigned to an action within a workflow with an additional “read only” attribute. This allows you to specify accessibility levels: invisible (attribute is not an action attribute) to visible (attribute is an action attribute but set to “read only”), editable and required.

Reference Types

These can be edited via the all new relation editor from within the entity view. Reference types become important for the definition of migration paths, where the domain designing developer defines how relational migration issues should be resolved upon deletion of an instance.

Sclable Domain Designer: entity editor tabs. This image shows the workflow-, attribute-, relation- and agenda editor tabs of the entity editor. The ? key will show a context sensitive hot keys cheat sheet, alt+h displays inline hot keys as seen in the screenshot. Our goal with hot key support is to allow for domain modelling without using a mouse. Image: "Sclable.com"

Sclable Domain Designer: entity editor tabs.
This image shows the workflow-, attribute-, relation- and agenda editor tabs of the entity editor. The ? key will show a context sensitive hot keys cheat sheet, alt+h displays inline hot keys as seen in the screenshot. Our goal with hot key support is to allow for domain modelling without using a mouse.
Image: “Sclable.com”

Hot Keys

The ? key will bring up a hot key cheat sheet, alt+h will display contextual hotkeys inline. Domain modelling now can be done from the keyboard without using a mouse. We will subsequently extend the set available hotkeys to in-graph editing and navigation as well.

Upcoming Features for 2.1

Subsequently we will add every Sclable Core Engine Feature to the Domain Designer. Depending on the complexity of the functionality itself it might take one minor release’s time for a core feature to be fully implemented by the Domain Designer. For the 2.1 release we plan to implement the Agenda/Action Binding Editor and the Selection Editor.


Sclable Core Engine 2.0

By Michael Rutz,

The Sclable Core Engine as of platform version 2.0 has been re-written from scratch. For two main reasons: first, to have a less monolithic architecture and second, to allow developers to extend the Sclable core.

At the same time we added some new features we’d like to introduce. Read on to learn how Sclable 2.0 adds more performance and security to your applications and lets you utilize events and triggers.

Postgres Performance Comparison from versions 7.4 to 9.4. One of the reasons why Sclable uses PostgreSQL is because of its continuous performance improvement. With the new Sclable Core 2.0 we are getting more out of the PostgreSQL community’s endeavours. Image: "PostgreSQL Addict"

Postgres Performance Comparison from versions 7.4 to 9.4. One of the reasons why Sclable uses PostgreSQL is because of its continuous performance improvement. With the new Sclable Core 2.0 we are getting more out of the PostgreSQL community’s endeavours.
Image: PostgreSQL Addict

Higher Performance

This one must go to the beginning of this post: The main challenge before the rewrite was the varying performance for agenda filtered views of an entity. Even though we aimed one step further to filter on an attribute level, we wanted to increase performance. We solved that issue by tying the Sclable user management to the PostgreSQL user management. Instead of primarily operating with filter functions, Sclable uses real views, thus allowing PostgreSQL to plan the execution of the query more efficiently and increase the performance.

Higher Security

With tying the user management to the PostgreSQL roles, the Sclable Core Engine is secured down to the data level. The users are not only logged into the application, but into the database as well with the very same privileges. Hence the users have the same level of access on every architectural level of the Sclable Business Application Development Platform.

Event and Trigger System

The event and trigger system introduced with version 2.0 allows you to extend the Sclable SQL core with additional functionality by simply hooking SQL functions into events and triggers. This reveals a world of great possibilities from process automation to arbitrary business rules.

Onion Architecture

Sclables Core Engine architecture has been redesigned to collect features in modules. Every module is stacked on top of the modules it depends on, just like layers of an onion. From the domain modelling point of view, the Sclable Core Engine modules belong to the domain «Sclable» and can be read and understood just like any business domain modelled with Sclable, thus being a good example of how to structure a domain.

Sclable Core Entities. This Graph shows the dependencies of the entities of the Sclable core domain grouped by modules. An interactive version "can be found here". Image: "Sclable"

Sclable Core Entities. This Graph shows the dependencies of the entities of the Sclable core domain grouped by modules. An interactive version can be found here. Image: Sclable

System, Event and Builder

The minimal core consists of the system module containing domain, module, entity and attribute information and logic. Followed by the event module, needed to execute the appropriate actions upon changes to the instances of entities in the system module. Those actions reside within the builder module.

Data Integrity Modules

‘rule’, ‘restriction’ and ‘path’ are the next Sclable core modules in the hierarchy. It’s all about data integrity. ‘rules’ are regular expressions and operators applied to attributes. The ‘restriction’ module describes uniqueness and requiredness of attributes and handles references between entities. The module ‘paths’ chains references to more complex relations such as ‘customer’ → ‘city’ → ‘country’.

Reference Types

To give a better description and control to the behaviour of references, we introduce reference types. In a relation of ‘order line’ → ‘order’, the type would be ‘cascade’ (which is the default), allowing to remove all order lines of an order when the order is deleted. While this is a good thing to keep the order lines entity clean, it would be catastrophic removing all customers for a city when it gets deleted. By describing such a relation as ‘restricted’, a city can’t be deleted as long as there are customers pointing to that city. ‘decouple’ will leave instances with the former reference set to NULL and ‘delegate’ will force users to point to an instance to be referenced, replacing the former reference.

Workflows with Action Triggers

Workflows became a system module of it’s own to describe the lifecycle of an instance of an entity. One of the most important new features with workflows are action triggers. action triggers allow you to execute functions on the database before and after a specific action has been executed on an entity. They present the ability of a more fine grained hooking than a trigger on the entity itself which is executed on every change.

Delegation with data filters on attribute level

One of the top onion layers of the Sclable Core Engine is the delegation module, responsible for authentication and authorization. Sclable 2.0 implements an authorization down to the attribute level: Users have access only to the attributes of an entity they are  allowed to read by the actions they are allowed to perform.

A new Action Binding Concept

In Addition to this, Sclable has a new action binding concept as of 2.0. Take a Sclable based CRM for example: A user is responsible for customers in Germany and Austria. The entity customer has a relation to the entity city, which is related to the country. The agenda ‘key-account-manager’ allows to execute the edit action on customers bound via the city to a country. When assigning the user to this agenda, we select the countries Germany and Austria to grant access to all customers in these countries.

Sclable Selection use case example. This image shows a simple use case for Sclables concept of “Selections”: For the display of chosen manufacturers from a business domain only the name of the company along with the latitude and the longitude of that company’s city is needed. It is a simple task of click-selecting attributes along a chosen relational path. The Sclable Core Engine automatically writes the appropriate SQL join statements for the auto-generated database view which ultimately is filtered by the user’s scope. Image: "Sclable"

Sclable Selection use case example.
This image shows a simple use case for Sclables concept of “Selections”: For the display of chosen manufacturers from a business domain only the name of the company along with the latitude and the longitude of that company’s city is needed. It is a simple task of click-selecting attributes along a chosen relational path. The Sclable Core Engine automatically writes the appropriate SQL join statements for the auto-generated database view which ultimately is filtered by the user’s scope. Image: Sclable

 

Selections

Selections were  introduced in version 1.0 as an extension to the core. Selections represent a set of values from attributes or calculated/aggregated attributes throughout a defined relational path. In version 2.0 we decided to give them more importance by adding them as a core module. They now integrate well with the authorization concepts of the delegation module.

Internationalization

Last but not least, we refactored (and simplified) Sclable’s built in data dictionaries. Translations can be activated for attributes since 2.0. Translations then are stored by entity, language and instance, thus reducing the amount of queries to translate an entire instance dramatically.

Recap and upcoming features

With its rewrite, the Sclable core is ready for our proposed roadmap. The onion architecture allows developing and releasing of feature packs every quarter as well as the maintenance of legacy versions. Amongst things we already queued for the upcoming version 2.1 are re-written calculated attributes along with a concept for aggregated attributes, both available within your selections. Keep an eye on our roadmap to stay up to date!

 


Business Logic and the Relational Database

By Michael Rutz,

Martin Fowler has written an article titled “Domain Logic and SQL” on the topic in February 2003, the same year Eric Evans published “Domain Driven Design – Tackling Complexity in the Heart of Software”. Recently we were asked (again) where we would see our position. Obviously, we favour an architecture where the relational database holds as much logic as possible, but why? And how does Sclable come around the common pitfalls?

OOP (Object Oriented Programming) versus SQL

The debate is largely run by technicians divided into two camps: OOP and SQL developers. Unfortunately they’re often divided by organisational and cultural splits. The OOP party isn’t allowed to apply changes to a database as they require, while SQL architects refuse to write helpful views for maintainability and performance reasons.

Though living on both sides, Fowler himself admits to be biased towards OOP (or at least, was in 2003), but he states in his article: “By hiding SQL, application developers are excluding a powerful tool”. To give an impression on how intense discussions already became back then, Fowler mentions that he has been run out of a company by a former client’s OO expert group because he was a “data modeler”.

In this post I roughly follow the route Fowler takes in his article. While written in 2003 and a lot of people still referring to it, many things have changed. I’ll show how Sclable takes advantage of the new world of RDBMS.

Complex Queries

Many application developers would still shy away from even minimal complex SQL queries. This is absolutely understandable, as SQL differs in many ways from programming languages. And when it comes to  JOINs over ten or more tables along with calculations, aggregations and windows, nobody would say, it’s maintainable. How would you be able to react on domain model changes?

Sclable’s concept is called “Selections” and puts an end to this dilemma. Selections can be built extremely fast with a visual query builder generating user permission aware (!) views. Change your domain model and your Selections morph with your requirements.

Transaction Script, Domain Model and Logic in SQL

Fowler compares three approaches to query data and apply domain logic: A procedural pattern called “Transaction Script” described in his book P of EAA with the advantage of having business logic and loading separated in your code, the object oriented domain model with the advantage of a reusable data loading and a more separated yet encapsulated business logic. Both have the downside of requiring multiple queries to lift more data into the memory as required for the logic. The third approach is written in pure SQL. It is quite obvious that the latter is the most powerful, performant and efficient way to go for.

Looking at Performance

Fowler sees more priority in writing maintainable, reusable code than in thinking of performance first. This sounds reasonable, but he also admits that his SQL query example performs 20 times faster in the test setup of his “little laptop” than the two other approaches. His suggestion is to enhance OO code later on, where necessary with more complex SQL queries. Obviously you’ll end up having both: maintainable but unperformant code at large and a growing base of SQL painfully hard to maintain.

Sclable models the domain directly in SQL and adds APIs giving you the benefits of all three approaches: You can extend your business logic in an OO manner in the language of your domain, you get an automatically generated metadata mapping to achieve that and all will be built on top of the full power of SQL.

Modifiability

Fowler sees modifiability as the main reason why developers put business logic in memory rather than into the database. He sees two limits in the capabilities of SQL: You can’t use non-standard extensions if you want portability and you might need some business logic for your session data before persisting it in the database. The next five sections will go into detail about this.

Understandability

This is a topic we have been dealing a lot with at Sclable. Modelling relational database structures and writing business logic are often two different disciplines, since databases are seen as a persistence layer while business logic is made to work on data. Database architects refuse to tackle domain logic, and on top of that, programmers have a hard time to understand SQL to an extent required for writing business applications.

At Sclable we aim to visualize structural as well as logical concepts so anybody, even the end user will be able to understand. Furthermore, we provide APIs on every level of the platform. Our Low-Level SQL-API translates domain logic to our PHP-API which translates it to your application front end. The generator component of the Sclable Core Engine keeps your domain logic moving and morphs all dependant layers with it. Think of it as WYSIWYG on application scale.

Duplication

Fowler states that duplication easily can be avoided in an OO domain model approach. To avoid duplication of SQL code, you would have to create views in the database. In some environments, this would move it out of the scope of the application developers. Well, we use views in Sclable extensively. A fairly large Sclable Business Domain with more than a hundred entities can easily have about the same amount of calculation and aggregation functions and four times as many views to produce domain logic specific results. And all of it only in the database in SQL.

Instead of avoiding a powerful tool to prevent duplication, we are generating all required views and the Sclable Core Engine with its builder component modifies them upon domain model changes, so they are free of maintenance while your domain model always stays DRY.

Encapsulation

A common approach in OOP design is to encapsulate logic. The application layer is having modules interacting via interfaces to simplify their complexity on the outside. Yet the database is seen as one big module a layer underneath. Fowler argues that changes in the data structure otherwise would cause a large ripple effect across a system.

At Sclable, instead of separating the application from its data, we move the domain logic into the database. We still encapsulate logic in modules, but give them a place of its own beside their data. The system overcomes the painful round of editing across the application by listening to the model changes and executing necessary changes automatically.

Database Portability

A big point to not utilize the database all too much is portability. Using the strengths and powers of SQL makes your business application depending on a specific database vendor.

This is true. But as with every business application it depends on the project if this issue has to be taken into account all too much. How likely will it be that you will need to change your database vendor at all? Sclable ships with PostgreSQL on board. And that’s for more than one reason.

Testability

Although it is common practice in SQL not to test, at Sclable we have written our own SQL testing framework we named “Sclunit”. Using the advantages of transactions, we finally are able to run tests directly in the database, with the business data and its logic in a completely non destructive way.

Summing Up

Fowler suggests to go for the development team’s personal comfort. And he is still true, when he states that a lot of people find SQL terribly cryptic. For any critical performance issues you might have to write queries nevertheless. Our experience is that it will be very likely that you’ll end up with an incomplete domain model having parts of your domain logic in SQL as soon as your model gains complexity.

At Sclable we made it our goal to put an end to the debate whether to go for in memory only or use the power of SQL (structured query love). We’ve been set out to deliver a Business Application Development Platform that gives you the best of all. For the majority of business logic you’ll actually never need to write SQL at all, Sclable’s Core Engine does that for you.
Sclable’s development stack also allows to seperate modelling from coding. Coders do not need to bother the system’s architects when it comes to write boring, repetitive SQL and the system architects can focus on the domain model.