Towards own product. From custom development to own platform
I'm a developer ASP.NET. I have a small development team. In this article, I will touch on some of the problems of custom development and how we solve these problems, and how we gradually moved to creating our own product-a web platform with an approach where SQL is at the top of the list.
Key problems of custom development in our practice
When making another project according to the custom development scheme, when the code is written for the project on a certain framework, you face the same set of problems.
The customer's expectations for project deadlines always exceed the reality. The full development stack involves the interaction of many people, micro-delays occur, and as a result, it all stretches (even without taking into account the debugging period).
In such a scheme, the programmer is the narrowest link, and therefore this link should work at full capacity without unnecessary downtime.
Developing from scratch and a full stack means more places where you can make mistakes. This is especially true when trying to «speed up» the project. New programmers are added, and some decisions on requirements are made rashly, right on the go.
As a result, this leads to errors, long debugging and unnecessary nerves.
Debugging is stretched due to the fact that the error can be located anywhere in the stack.
Very often, the customer compares custom development with the creation of a similar product on a ready-made system-a CMS. This does not take into account flexibility and the possibility of further development.
Custom development is always expensive, because the number of people required in this process is much greater than in the assembly of ready-made modules.
At the same time, the qualifications of these people are required to be higher than when working with CMS modules in general.
The human factor is increasing
Incorrect data access and giant implicit cycles in the code can lead to productivity problems. It is difficult to search for such errors in the project when they can be made at any level (in storage, business logic, controllers).
Every developer should have a fairly good understanding of these points, otherwise you may get performance problems somewhere in the project code in the future.
At the same time, training a single programmer on a full stack is quite a difficult job. And it is not immediately that performance issues become relevant for him.
If such a specialist decides to leave the team, this can greatly affect the future quality of the project. And here it is difficult to change something, because in any case, there are people who know more about the project than others and no documentation can compensate their loss for the project.
In our case, custom development is not sugar.
Perhaps there are some teams that make everything easy and simple (well, or support this feeling for a potential client).
If these problems are imposed on the client’s unstoppable appetites, the picture turns out to be quite sad. It takes a long time, it is expensive, we make a project with errors, we redo it to meet new requirements-the client's wishlist, until the budget ends and the project is frozen for an indefinite period.
These problems led us to our approach, and it was necessary to change something in it, perhaps to abandon something. Especially, of course, the situation is depressing when relatively typical things (for example, adding a field for editing to a table) are done for a very long time. We needed a new way to make these edits easily and without the hassle of various specialists.
What does the customer need?
What does the customer want from us?
One of the main factors is money. He wants to save money and understand his future project development costs.
I think for most custom projects, the cost of changes is gradually increasing. In the beginning, we move quickly and implement all the necessary features. But as you use the project, changing / adding features makes it more difficult/more expensive.
Speed of implementation of new features
Speed of implementation goes hand in hand with savings. Anything that takes a long time to implement and requires more budget.
A clever customer instinctively understands that they do not need a long, heavyweight project in which they can get stuck for a long time. You need to do something small quickly, implement it, get feedback (and collapse the project if there is no result).
If you do a big project for a year or two, then you need to invest a large amount, and there is no guarantee that the plane will fly.
The product can be changed quickly, rather than waiting for each feature to be implemented for weeks.
The project may contain a lot of hypotheses that the customer wants to test on their users. And it is important to keep up with the rhythm of the customer's wishlist. Yesterday we came up with it, today we described it, tomorrow we'll implement it and use it. Thus, small improvements should ideally be introduced.
Quite ideal is when the customer can fix some points in the system in a hot way (of course, with understanding and full responsibility for their actions), as we do in Excel, changing the format of tables or adding a new formula.
Looking forward, we assume that the customer knows SQL, which means that he can change a lot of things in our system if he wants.
It is for this reason that the customer chooses the development for their needs.
In custom development, you can do anything in any way.
This is the point where, by default, the client sets the highest possible requirements for custom development: unique design, all the requirements of the customer's business analyst, strong integration with external systems.
An important nuance is customization in the future for emerging business needs. If this is not taken into account at all at the beginning, you can get a good system that can not be changed in any way (very expensive), or the change is made through the giantsest crutches.
Low risks of changing suppliers
Ideally, for the customer, the project should not depend on the service provider in any way. This is achieved by using well-known proven technologies to have enough options to hire the right specialist. The second most important thing is the documentation that is created (or not created) for the project as the system develops.
Quality of the final product
During the start-up phase, the customer expects the product is well tested and error-free. In theory, yes. But look at the number of patches of the completed Windows product (the comparison may be incorrect due to the complexity and scale, but the essence is about the same).
Here it is more important to understand the iterative nature of the project, and to treat errors with understanding (found — fixed — working further).
In any case, the developer, for his part, should generally somehow reduce the risk of errors, without significantly affecting the budget inflating.
What the developer can offer: custom development and complete product
What can a web developer offer to such a customer?
By and large it has two variants:
- make your own standard ready-made solution for a specific task (for example, CRM) and sell this solution to the customer (if it satisfies their needs in full).
- offer custom development services (sell programmers' hours).
Both solutions have their advantages and disadvantages.
A ready-made solution is difficult to customize in the future, and custom development is very long and expensive.
The main dilemma of the customer is that you want to quickly get a ready-made solution, but at the same time have the opportunity to deeply customize and the opportunity to further develop our product.
Ideally, it would be good for a developer to make their own standard product and sell it to everyone who wants it. But there are nuances:
- it is expensive to create a product (you need to somehow make ends meet until the product earns money)
- it's easy to miss the market (no one needs it, it doesn't solve the problem, it loses a lot to competitors). Developers are not marketers, and most of them do not have a deep understanding of the end user's problems, focusing on other aspects of the product.
How to link ready-made products and custom development
Thus, it is understood custom work on fullstack is difficult to scale, low-profit, and rather thankless:
- it is quite difficult to accumulate a code base (it is not always possible to develop a component in such a way that it can be reused).
- errors can come from anywhere
- it is difficult to teach a novice developer the full stack. At the same time, it still leaves sooner or later.
- even simple edits take a very long time to implement.
Ideally, the solution was seen as follows:
- create a platform that allows you to make quick changes to the business logic without having to update the site code.
- a pool of standard components that can be managed from SQL. Everyone knows SQL, stored procedures in General have quite good capabilities for describing business logic (this probably sounds crazy for OOP programmers).
- reduce the development stack as much as possible so that the developer doesn't need to learn 6-7 technologies to support the project. Less technology-cheaper support, faster training, more reliable solution.
- standardization of elements and approaches to development. Previously, there was a problem-every developer did some elements a little bit, but still in their own way. And as the project develops, this makes it much more difficult to support the code. Ideally, the system should look as if it was written by one person.
- it is extremely important to retain the possibility of deep customization. Without this, most potential buyers of the platform will refuse due to the unavailability of a particular feature. In other words, the task is to make it possible to develop the system for themselves by relatively simple means, without fear that we will not be able to implement some business function of the application.
- the ability to increase the system's capabilities from project to project. Not in terms of taking other people's work on business logic, but in terms of polishing the capabilities of standard components (the appearance of options, editing complex bugs, etc.)
- fewer people on the project. The more developers you need, the more difficult it is to coordinate, and the more calls, meetings, and business logic explanations you need. As the number increases, the quality of the code will drop. And not the fact that this will somehow speed up the project.
How did the idea of the platform come about and what is its essence?
Managing metrics via SQL
In our accounting system, which we made for our needs, there was a Yandex. Metrica module.
I implemented an approach where a metric is just a stored procedure in SQL that is visualized in a table. The table can have nested tables that are also created via SQL.
As a result, you don't need to change the app code or update the PROD version to create such metrics. Changing data in tables, creating procedures, and checking on the site.
Nested metrics https://falcon.web-automation.ru/tst-customers
An important point here is that metrics are created gradually (now there are more than 250 of them, and most of them are no longer relevant). Thus, this approach saved a lot of time (compared to if it was done through the application code, for example, through LINQ, EF).
The first platform components
In parallel, we had a JS component table, which allowed us to quickly build a table for managing an entity (CRUD): we configured JSON, created a Controller method, prescribed methods in business logic and DAL, and the table is ready.
I came up with the idea to repeat the SQL approach on tables:
- the page contains a certain component markup (snippet).
- JS sends a request for settings to the database (where the database table stores information about the table settings)
- calling the required stored procedure (which returns data and additional settings)
- visualize all this on the page
The component table.
This is what will form the basis for building any platform component in the future:
- Any component is a markup snippet.
- It is transformed by calling stored procedures in a specific format into a Table, Form, or Dashboard, and so on.
- Any action in the system is a call to a specific stored procedure, which is generated by an example procedure. Each procedure has a hard output format that specifies settings and data on the component operation (there can be from 1 to 10 select in one procedure).
There was no idea to create a complete platform yet.
There was a great desire to speed up the creation of tables as much as possible and reduce the number of stupid errors all over the stack.
Instead of 3-4 hours to spend on it 30-40 minutes.
As a result, we solved this problem, but did not stop there and tried to implement other typical elements. The main components are static pages and forms.
At that time, the key problem on the way to the possibility of creating a platform was the work of forms. Initially, there was a workaround: give the platform as a revision project in the form of a solution on ASP.NET. But in this case we got upgrade problems and all the disadvantages of custom development on the full stack.
After adapting the forms to the new approach with SQL management, it became clear that we can solve everything else by analogy: uploading documents, resource Manager, dashboards, chats, etc. The gradual creation of platform components has begun.
Starting working on the platformm
At the same time, we also created client projects based on the new platform
There were more narrow questions: implementation of SignalR, integration, customization of appearance. We solved all these new needs for the customer's project. But we also added to the collection of solutions for our platform, as we expanded its capabilities and made it more flexible.
At the same time, the main idea was not to overload the platform itself with some narrow elements that were tailored only to the needs of a specific customer.
In other words, a certain mechanism was created in the platform that can be controlled via SQL, and this very SQL was already written for the project.
The result is a relatively stable core that does not change for the project and all major changes are made through stored procedures, which in general allow you to implement almost any logic.
Emerging issues and solutions
Two key issues at this stage:
- how to make a universal API while following the platform approach
- how to call actions such as sending SMS, clearing the system cache from SQL procedures.
We made 2 subsystems for the API: incoming API methods and outgoing requests to external APIs.
Incoming methods are, in fact, a controller method that accepts all requests, and, by the action code, defines the stored processing procedure. This procedure processes the input data and returns the result, which is given to the requester in JSON, XML, or plain text.
With outgoing requests, everything is a little more complicated. First, you need to prepare the request to the outside (this is one stored procedure — request), and then process the response from the outside (this is another stored procedure — response).
You can send requests by POST, GET, parameters can be passed in the body, header, or url.
Managing of API
To call non-standard actions, I first tried, but refused to implement assemblies in SQL Server. This is cumbersome and difficult to port, plus it requires additional external settings and creates dependencies on the external environment.
As a result, it was decided to implement the concept of External actions: the platform expects some additional select procedures from stored procedures, in which the necessary commands are passed, for example, send SMS.
Thus, we perform actions in SQL that are actually performed after the stored procedure is executed.
In general about the approach
The key idea of the platform is the ability to customize business logic while keeping the general approach (SQL procedures declaratively manage everything).
The internal code of the platform does not change for the project, only the SQL code is created for the specific business logic of the project. This makes it possible to update the platform without having to somehow combine custom design improvements (frequent changes) with core improvements (rare changes).
This approach also made it easier to create standard solutions. In our case, any ready-made solution is, in fact, just a large piece of SQL code (table definitions, stored procedures, SQL functions, and table data).
We have created additional tools inside the platform that make it easier to transfer parts of the project to another database in the form of SQL scripts.
We gradually create our own standard solutions and offer their implementation in working projects (for example, a Blog component or a knowledge Base).
Some things had to be left out. For example, strong atypical customization of internal pages in terms of design.
Landing pages in the system can be any (they are made so that the landing code is simply added in its entirety and adapted in the platform).
Customization of the appearance is possible through theme settings, but still has limitations. The main menu is located on the left, the upper bar has a standard view for the platform.
Examples of dashboards (indicator panels)
Highly custom projects for appearance-this is not about us. We give you more flexibility in terms of changing the business logic and a good default appearance.
In my experience, it just so happens that projects with high design requirements are the most problematic and highly risky.
Design and visual appeal are important, but for business systems, I don't see a particular need for design uniqueness. These are rather personal characteristics of the product owner that do not benefit this product itself.
In general, we sacrificed some flexibility in terms of design, but left the possibility of changes via CSS and JS.
In this case, the central workspace can be collapsed as you like. The Form component allows you to use the default layout of elements, but also allows you to implement your custom markup on Bootstrap 4.
Custom markup for the form
To fully master the system, you need to know 2 technologies: MS SQL and Bootstrap 4. SQL provides us with the plasticity of business logic, and Bootstrap provides the flexibility of data output in the interface (you can do without it by default).
Why did we choose SQL? Developing your own language is difficult and it is not clear why, and the developer will have to learn something new for himself. According to the subjective feeling, most programmers know SQL in one way or another (well, at least, they encountered it in high School).
For algorithmic tasks, SQL allows you to perform basic actions: conditions, cycles, and typical functions.
SQL — a tool for extracting data from a database. Using it as a basis, we at least reduce the possible risks of performance problems (of course, this is not a panacea, but it is much better than Lazy loading and unoptimized queries in LINQ).
Components also return multiple selects in a single call to a stored procedure. This also reduces the time required to process requests, and you need to open fewer database connections.
No heavy ORM is used internally (for example, Entity Framework, Linq2SQL). All queries are written in SQL and wrapped in objects via Dapper (which is comparable in processing speed to DataReader).
The main development is carried out through the administrator's office.
You can potentially make hot edits via your phone (the interface is adaptive for mobile devices).
Managing page settings
Of course, a full-fledged IDE has its big advantages. We are gradually adding various features to our sql editor (code highlighting, test run of stored procedures, output of launch parameters, full-screen mode, etc.).
How the implementation of the platform affected projects
We have reduced the number of people on projects. For the project, we employ no more than 2-3 people (previously there were up to 10 people). This simplifies control, reduces unnecessary interactions on the project, and reduces the cost of work.
An extremely important point is that the platform is polished with each project.
Sometimes internal errors are detected during the project implementation. They are corrected, and these edits improve the quality of not only a specific project, but also other projects made on the platform. Gradually, new features and customization settings appear, which can be used in a unique way in each project (because they are changed through stored procedures).
This is one of the main advantages of developing your product compared to developing without your own-you continuously improve its quality along the way. The solution to each system problem is not lost, but integrated into the product.
The fewer places where a developer can make a mistake, the faster you can find this error. In our case, 90% of errors are inaccuracies in SQL. This is where you first need to look for the problem. This also affects the debugging time of the solution and, in general, the timing of the stage.
On projects, one of the tasks is to identify inaccuracies in the project (appearance, business logic) and work out these points should be solved at the platform level, so that it is possible to use simple tools (CSS classes or settings in the CP) to solve these points in the future. However, this leads to the side effect of bloating the documentation, since each such setting requires a description of its application.
An additional advantage for us is easier training of new developers for the platform. If a person knows SQL and has heard something about HTML, then in general it will not be difficult to train them to work on the platform. Previously, full stack training could take 3-6 months. Now, in general, with a thorough study and practice, you can master the main features of the platform in 1-2 weeks.
The cost of work decreased.
In most cases, most of the data output concerns are hidden from the developer. If you don't need special customization, you can build solutions very quickly and it doesn't require a lot of money (especially if the person already knows the nuances of the platform well).
What has changed from the customer's point of view?
Subjectively, customers have fewer complaints about inaccuracies and improvements to the appearance.
Many points are made by default, the developer does not need to think about the small details of the table output. Its task is to configure the stored procedure correctly, and the platform ensures that the table works in accordance with its settings in stored procedures.
The cost of changes has decreased. Changes one person makes, there is no need to use complex cycle updates to the business logic. If the customer knows SQL, then he can change some elements in the system himself.
The customer initially sees the demo and tries it on his project. Its appearance expectations are formed at the earliest stage. And there is no strong discrepancy in the future between the desired and the actual.
Fewer resource requirements. In general, the platform can be put on Win hosting for 5$ per month and this is quite enough at the initial stage of project development (and then it all depends on traffic and the amount of data in the database). In custom fullstack development, we always used VPS (and memory costs were higher, apparently due to suboptimal places that sometimes happen).
The process of continuous improvement is addictive. You gradually get used to the process of implementing and polishing components.
We still have a lot to do in terms of adapting the developer to the platform, improving documentation and developing system tools (DB diagnostics, stress testing tools for stored procedures, etc.).
At the moment, we are working on connection tests with other databases, so that we can process data in the platform from non-native databases (PostgreSQL, MySQL, Oracle).
Another important area is working on Linux (in general, there are already demo stands on Linux, but they still need to check all the features of the platform). Now basically the system used on Win hosting or VPS Windows Server.
And most importantly — is creating the business layer modules. I. e. is already integrated units, focused on business objectives — personnel records, report cash flow statement, bill generation and other components that in the end, going in the subsystem.
Interactive report cash flow statement
In a new project, you can not create a certain subsystem from scratch, but take our existing block as a basis (for example, the cash flow statement report) and further adapt it to your needs. Any component created on the platform is an SQL package that can be moved to a new DB.
One of the demo cabinets
Disadvantages of our approach
This solution has the following disadvantages:
- some features are better done in the backend language, rather than in SQL (for example, calculating the MD5 hash for Cyrillic in SQL Server).
- there are still limitations: in terms of functionality (in terms of integration, for example, file transfer via the API), and somewhere you have to look for compromise solutions.
- still, the browser is not a development environment (many people prefer to work in the IDE, rather than in the cabinet on the site)
- potentially, there may be needs that are better solved in C# than in SQL in terms of CPU optimization (in this case, they will need to be solved either by an external service, or by implementing a new external action in the core, which is not good from the point of view of architecture in general).
- rash edits from your personal account can lead to errors or instability. These are the risks that arise-a fee for the speed and ease of edits based on the application's business logic.
We are still at the beginning of the way.There is still much to do.
But we can already say with certainty that we have our own product, which we gradually Refine with each new project or our standard solution/demo.
Originally published on: our page on vc.ru https://vc.ru/tribuna/173182-na-puti-k-svoemu-produktu-ot-zakaznoy-razrabotki-k-sozdaniyu-svoey-platformy
How do I know the budget / timeline for my project?
1. Create a project concept
2. Send us your concept paper
3. We will prepare a commercial proposal with details by modules
Falcon Space Platform
This is a reduction in the cost of ownership
at the expense of fewer people to support
This is a quick change
while using the program
This is a modern interface
full adaptation for mobile devices
If you like our articles, then please subscribe to our channel in Telegram - Falcon Space.
In it we will publish updates on articles and other materials regarding our platform.