Menu

Dimensional Data

From the blog

How we built a Stack Overflow Community questions analyzer (and you can too)

Noutați

Being part of the GitLab collective is an opportunity to learn first hand about the challenges the community using the DevOps Platform is facing. As a Collective Member logging between 2-3 times a week in StackOverflow reading the questions and discussion posted about GitLab and manually sorting them by ‘Recent Activity’, ‘Trending’ and using Dates, I asked myself: how can we leverage this wealth of data and discover feedback, while finding the most frequent topics where the community has questions? This would be an opportunity to get a quick overview of topics where the community regularly needs help; this would also make it easier for us to create relevant content for them. Manually sorting and extracting the text of each question wouldn’t be sustainable, so creating an automated way would be the most efficient way to proceed. Experimenting with data-oriented content creation Finding out what the community is working on, and what they need help with while using GitLab, can help us to create better educational content that could expand their understanding of GitLab. To achieve this goal, the solution I created after a few iterations is depicted below: Where the Bill Of Materials consists mainly of: GitLab DevOps Platform Stackoverflow API Kubernetes Cluster Open Source Python libraries: scikit-learn (TF-IDF) Streamlit (front-end) Spacy I leveraged the GitLab DevOps Platform to organize the projects using groups: The Loader project pulls questions about GitLab from the StackOverflow API, pre-processes the text and makes it usable for a second project: a Visualizer to create customized dashboards. The automated process executed using the DevOps Platform is outlined below: Pull data from StackOverflow API Preprocess the response extracting relevant fields from returned JSON Build a corpus and calculate TF-IDF Scan for security vulnerabilities Review Application and display its resulting dashboards using Streamlit Deploy the built application to a Kubernetes cluster Loader and Visualizer projects have their own codebase and pipelines, which is helpful if different teams need to work separately on them. However, one project can require the other, which raises the need for cross-project automation. This scenario means a multi-project pipeline is useful to automate the whole process. The multi-project pipeline enables use cases such as: As an NLP Developer I want to work on the NLP Pipeline in the Loader Project and automatically trigger the creation of a new visualization As a Streamlit Developer I want to work independently in the buttons and data visualization without touching any NLP Pipeline backend The outlined process above is automatically run defining the steps in a multi-project pipeline sharing artifact: Finding the most frequently occurring words The Feature Engineering step will help me to analyze the text in the whole dataset of GitLab questions. Using a simple yet powerful technique – TF-IDF – we aim to find the most relevant terms utilized by the community. By using this technique in the pipeline execution, I represent words in numerical values and later rank them in order of importance. This approach serves as a baseline for further improvements. More detail about this algorithm can be found here. Did we achieve any success? One run of the multi-pipeline in our solution results in dashboards such as this one: As an end-user of these dashboards I can immediately conclude that the main source of questions are around GitLab CI, pipelines […]

Read More

Two sizes fit most: PostgreSQL and Clickhouse

Noutați

Since the introduction of System R in 1974, relational databases in general, and SQL databases in particular, have risen to become the dominant approach to data persistence in the industry, and have maintained that dominance despite various significant challengers. Though some have rumored the death and decline of traditional relational databases, PostgreSQL has turned out to be an improvement on its predecessors, as well as its supposed successors. In fact, the open-source MySQL database was so ubiquitous that it became part of the eponymous LAMP stack (Linux, Apache, MySQL, Perl) that dominated early web development. The one big exception to this trend is OLAP, where specialized techniques that can drastically improve the performance of certain workloads have met with use-cases that actually require these techniques, with new contenders such as Clickhouse enabling qualitatively different approaches to analytics. One size does not fit all As often happens when a technology becomes dominant, it gets applied unthinkingly even when it may not actually be appropriate, and so all kinds of data was and is being pushed into general-purpose relational databases. Extreme examples could be found, such as developers creating remote Oracle databases for data sets with a total of 5 small elements (not columns, pieces of data) or Apple pushing their system logs into an SQLite database (a mistake they later corrected). Bind10 development started under the premise to solve scaling issues with Bind9 as DNS nameserver, using SQLite as backend. The DNS development was discontinued by ISC in 2014, and the OSS project Bundy remains inactive. PowerDNS focussed on performance scaling with MySQL/PostgreSQL early. In 2005, Michael Stonebraker, database researcher behind Ingres and later PostgreSQL, together with Uğur Çetintemel, penned a paper “One Size Fits All”: An Idea Whose Time Has Come and Gone arguing that this had gone too far too long and backing up that argument with benchmark results. In short, there were many workloads outside of the core application of databases, Online Transaction Processing (OLTP), where the general database architectures were outclassed sufficiently that it did not make sense to use them. It should be noted that Stonebraker and Çetintemel argued not against relational databases or SQL, but against a specific architecture descendent from the original System R and Ingres systems that were and still are being used by most general purpose database systems. This architecture has the following features: Disk and row-oriented storage and indexing structures Multithreading to hide latency Locking-based concurrency control mechanisms Log-based recovery In addition to special-purpose text indexing, the primary use-case for which the traditional architecture was proving inadequate was data warehouses, for which column stores were proving 10-100x more efficient than the traditional row stores. Clickhouse The prediction that OLAP database engines would split off from mainstream databases has largely come to pass in the industry, with OLAP databases now being a significant category in its own right, with vertica, the commercial offshot of the original cstore discussed in the paper, one of the major players. The practical advantages of these databases for analytical work are, as predicted, substantial enough that having a separate database engine is warranted. Or even necessary, as was the case for Yandex’s clickhouse OLAP database, recently spun out into a startup that just received a US $250m series B. The clickhouse developers wanted to have realtime analytics, […]

Read More

Customizing performance metrics in the Unity Profiler

Noutați

For more information, please see the Module Editor documentation. Profiler module API The Profiler module API allows you to add your own Profiler module to the Profiler window for all users of a project or package. When a Profiler module is defined in a project or package using this API, it automatically becomes available in the Profiler window for all users of that project or package. If you are an Asset Store publisher or a package developer, you can now distribute custom Profiler modules with your package. When a user installs your package, your Profiler modules will automatically become available in the Profiler window for your users. This enables you to expose your package’s performance metrics directly in the Profiler window. Several teams within Unity have already been using this API to distribute custom Profiler modules with their packages, including the Netcode for GameObjects, Adaptive Performance, and Mali System Metrics packages. How to use it To add a Profiler module using the Profiler module API, create a ProfilerModule script in your project or package, as shown below.

Read More

What’s new in Cloud Content Delivery: Spring 2022 update

Noutați

For example, you can use Game Overrides to set up time-based changes to your game by enabling and disabling game modes with Remote Config keys. You can then enable new content using CCD, and adjust rewards and IAPs using Economy (beta). To integrate Game Overrides with CCD, install the Remote Config package, link your Project ID, create an Override to target content by using your CCD badges and content bucket, integrate Remote Config into your game code, and retrieve the appropriate assets from CCD. This integration gives you greater control over your asset delivery, allowing you to make more changes directly from the dashboard while keeping positive player experiences at the forefront. To learn more about Game Overrides, click here. Also, visit this documentation page to learn more about integrating Game Overrides with CCD.

Read More

Supercharge Your User Interface with Skia4Delphi

Delphi

What do Google Chrome, Mozilla Firefox, Chrome OS, Android, Flutter, and Delphi all have in common? They have accelerated graphics thanks to Skia. This high-performance, cross-platform 2D graphics library is taking the development world by storm. It works with VCL and FireMonkey on every platform Delphi supports. One line of code can make your existing FireMonkey applications faster, or take it further and unlock new features for breathtaking user interfaces. Skia4Delphi is an open source project by Spirit of Delphi Award winners and brothers, Paulo César Botelho Barbosa & Vinícius Felipe Botelho Barbosa. So how do you use Skia4Delphi and take your application development to the next level? Register now and join Jim McKeeth and Ian Barker in this webinar to find out! Come back after the webinar for details on the contest, more links, and the replay. Here are a few shorter videos while we work on getting the full replay. What is Skia, You Want to Ask? Imagine you are a software developer who needs to reach all the major platforms today. Your traditional choices are: Native Direct access to hardware Separate code for each platform Web/Cloud Mostly shared code Detached from hardware But Now We Have… FireMonkey Native Compiled – direct access to the platform hardware Multi-Platform – one code base for all platforms Component Based Visual Designer – maximize developer productivity How Does Skia Fit in Here? Introducing… Google Skia Google Skia is the open source graphics engine for Google Chrome, Android, Flutter, Xamarin, Firefox, and many others. It provides common 2D APIs that work on a variety of platforms, abstracting complexities in implementing low-level libraries it uses behind it, such as Vulkan, DirectX, Metal and others, implementing many optimizations and new features. Skia Platforms: All the most popular platforms in the world, such as: iOS (including simulator) Android (including simulator) Linux (main distributions) Skia Features About Skia4Delphi OpenSource Uses a modified fork of Google’s Skia Cross-Platform & Multi-Framework 2D graphics library for Delphi Drawing Focus on Quality & Performance The Brains Behind Skia4Delphi Two brothers from Brazil 2021 Spirit of Delphi Winners Paulo César Botelho Barbosa Vinícius Felipe Botelho Barbosa With the encouragement and suggestions of Ian and Jim Skia for Delphi ECONOMY – Open-source project, completely free EASY TO USE – Easy to install, simple code Compatibility Delphi 11+: All platforms Delphi 10.3+: Windows & Android Delphi XE7+: Windows Available via… Skia4Delphi Library The library conceptually exists in 3 parts: Skia API (Console, VCL, FMX) Access to the pure Google Skia library, through a single unit: Skia.pas Controls (VCL, FMX) TSkAnimatedImage: Play Lottie, Telegram stickers, animated GIF and animated WebP TSkLabel: Multiple styles in text, font weight, justify alignment, limit max lines, background color on parts of the text, auto size width and height, advanced decorations and more TSkPaintBox: use OnDraw event to draw with Skia API directly on control TSkSvg: load icons svgs, change colors and wrap mode App rendering (FMX) Optional feature that when enabled, the FMX graphics engine will be replaced by Skia4Delphi’s graphics engine, that is, the entire app, all the controls on the screen, will be painted internally by the Skia-based canvas. That is, by adding just 1 line of code to enable it, your entire app will automatically: Improve the quality of drawings; smoothing of jagged edges Gain […]

Read More

7 Undeniable Reasons To Love Your Python GUI

Delphi

Python is a high-level programming language that may be used to create a wide range of applications, such as online applications, software and game development, network programming, graphical user interfaces (GUIs), scientific and quantitative applications, and more. It also has a long list of success stories, demonstrating that it assists a wide range of businesses in accomplishing their objectives. This article will be looking at unquestionable reasons to love Python GUI. What are Python GUIs and the main reasons to love them? GUI stands for Graphical User Interface, and it refers to computer applications that allow users to interact with an underlying application or system through a visual interface. For example, our smartphones’ graphical user interfaces (GUIs) will enable us to engage with many functionalities via the display, which we can touch, tap and swipe on. In simple words, a Python GUI is a graphical user interface created in the Python programming language. Python is a widely-used programming language due to its ease of use, widespread adoption, and, most significantly, its beginner friendliness. Python is wonderful for creating graphical user interfaces, as well as being extremely valuable in the disciplines of data science and machine learning. It even provides a number of frameworks that beginners can utilize to get started with GUI development. What Makes Python Unique? Python’s codebase is clean and well-structured, making it easy for developers to update and maintain the software. In addition, they don’t require developers to write any more code, which saves them both time and effort. As a result, they may devote their time to something more beneficial for the company. Python’s syntax consists primarily of English keywords, emphasizing code readability. During the development of the application, the readability of the code is crucial. When the software is used in the real world, the customers’ needs may change. You won’t have to worry about shifting needs with Python. Even if the developers who designed the program earlier have left the company, the new developers will be able to read and comprehend the old code and apply the new requirements accordingly. Why is PyScripter the best IDE for Python GUIs? PyScripter originated as a simple IDE to supplement the excellent Python for Delphi (P4D) components by providing a reliable scripting solution for Delphi applications. Because it is written in a compiled language, it has a more current user interface and is faster than some other IDEs. It also has several features that make it a good Python development environment. This fantastic IDE seeks to create a Python IDE that can compete with other languages’ traditional Windows-based IDEs. PyScripter is an excellent program. Lightweight, versatile, and extendable with a lot of features. Because it was built from the ground up for Windows, it is substantially faster and more responsive than cumbersome text editors, general-purpose IDEs, or other Python cross-platform IDEs, making it a perfect match for Python GUI programming and application How does Delphi supercharge Python development? Python’s versatility as a programming language stems from the fact that it can be used to create for all major platforms. You can develop programs for not only Windows but also Android, macOS, and Linux with the right Python tools. Using development tools that allow you to do so opens up a whole new universe of possibilities for your […]

Read More

How To Achieve High Performance In Cross Platform Apps

Delphi, Noutați

Whether you are working on a small or large development project that requires you to deal with an overwhelming amount of code, you surely want everything to be as responsive as possible, especially if you’re developing cross platform apps where every single byte of code counts and memory is often tight. Everyone desires to achieve and experience high performance; in fact, this is often one of the reasons people choose Delphi due to its ability to produce truly native apps which operate without any intervening compatibility layer to slow things down. Interestingly, in this video, Primož Gabrijelčič will share some useful tips and tricks on how to achieve high performance in Delphi. What are some great tips and tricks for improving performance in our apps? Performance to people has different meanings. For some, it may mean programs are running just fast enough. For instance, if you are typing your codes in RAD Studio, you want the development environment to be responsive enough for every action you make. In other cases, some people simply aim for the raw speed of the program while others expect servers to respond to queries in a reasonable amount of time. In this video, Primoz will list down all the possible ways you can do to improve performance in Delphi. The first step is to always confirm the problem and measure the code. Once you have figured out where the problem is, the best way you can approach is to fix the algorithm to get the best performance enhancement. Another equally useful option is to fine-tune the code or add the so-called parallelism to your code. You can also take advantage of the external libraries which might solve your problem faster or you can also rewrite your code in assembler as your last resort. The video will also provide actual demonstrations of how these techniques are being applied to Delphi. Primoz will also show how running less code can impact the performance and some important things to consider when executing the program. To learn more about how you can effectively improve your performance in Delphi, feel free to watch the video below.  

Read More

Visualize your own data structure in Delphi

TMS Software

In April, we did a blog on how to retrieve Google Analytics results, and display them in a TMS FNC Chart instance. The data, coming from the request, was displayed in a points collection. Did you know that you can also map your own data structure? TMS FNC Chart is designed to easily integrate your own data with a couple of lines of code. This blog will cover this, but first, let’s take a look at a general introduction video on what TMS FNC Chart has to offer.  Introduction Data mapping For the purpose of this sample, we take our CARS.csv file, shipped with a couple of our FNC demos. The file contains a list of cars, with useful info such as the power, cylinder capacity and price. First of all, we load our csv data in a TTMSFNCGrid. TMSFNCGrid1.IOOffset := Point(1,1); TMSFNCGrid1.LoadFromCSV(‘CARS.CSV’); TMSFNCGrid1.SortData(1, sdAscending); TMSFNCGrid1.Cells[1,0] := ‘Brand’; TMSFNCGrid1.Cells[2,0] := ‘Type’; TMSFNCGrid1.Cells[3,0] := ‘CC’; TMSFNCGrid1.Cells[4,0] := ‘Hp’; TMSFNCGrid1.Cells[5,0] := ‘Cyl’; TMSFNCGrid1.Cells[6,0] := ‘Kw’; TMSFNCGrid1.Cells[7,0] := ‘Price’; TMSFNCGrid1.Cells[8,0] := ‘Country’; Next, we want to visualize the price for each car in a bar chart. First, we add a new series, and set the type. var s: TTMSFNCChartSerie; begin TMSFNCChart1.Series.Clear; s := TMSFNCChart1.Series.Add; s.ChartType := ctBar; s.XValues.Angle := 90; end; To visualize the data in the chart, we can loop through the rows in the grid, and manually add a point for each row, pointing to the data. But there is an easier way. To load data in the chart, mapping directly on the grid data, we begin by implementing the OnGetNumberOfPoints event. procedure TForm1.TMSFNCChart1GetNumberOfPoints(Sender: TObject; ASerie: TTMSFNCChartSerie; var ANumberOfPoints: Integer); begin ANumberOfPoints := TMSFNCGrid1.RowCount – 1; end; The data is requested for each index between 0 & ANumberOfPoints through the OnGetPoint event. In this event, we can map the index on the series data. procedure TForm1.TMSFNCChart1GetPoint(Sender: TObject; ASerie: TTMSFNCChartSerie; AIndex: Integer; var APoint: TTMSFNCChartPointVirtual); var v: Integer; begin v := 0; if TryStrToInt(TMSFNCGrid1.Cells[7, AIndex + 1], v) then APoint.YValue := v; APoint.XValueText := TMSFNCGrid1.Cells[1, AIndex + 1] + ‘ ‘ + TMSFNCGrid1.Cells[2, AIndex + 1]; end; As you can see, a couple of lines of code immediately gives you a first impression of your data. Further customizations can be done, but the chart will map on the data, even if it refreshes the next time your app starts. CSV is a small sample and of course, the data delivered to TMS FNC Chart can be of any source type. v2.0 Our team is working hard on the next version which will include a significant amount of new features and improvements related to data import, look & feel and out of the box experience. Stay tuned for more! Want to explore the capabilities of TMS FNC Chart, go ahead and download it from our product page.

Read More

Meet Deputy, the new IDE expert, part of RunTime ToolKit

TMS Software

Developing in Delphi sometime presents a frustrating work-flow, experts allow us to alter some behaviors and Deputy targets the one that I disagree with the most, prompt for overwrite. This post will focus on the developers daily tasks and introduce Deputy, a part of RunTime ToolKit, and explain integration of Marshal and Caddie. Introducing Deputy Deputy, an IDE expert, remediates orphaned processes to avoid the prompt for overwrite. By default the IDE does not check before running a compile, it runs a few seconds of precompile then prompts that it can not overwrite the target. This expert watches before that compile starts and clears out the old process before the compile starts. See it in action: Availability: Deputy is available in GetIt for Delphi 11, 10.4 and 10.3. RunTime ToolKit integration Deputy provides the ability to download and launch Caddie, as well as the Marshal demos for VCL and FMX. Future releases will integrate specific items from Caddie and Marshal to make the experience more seamless. Marshal Highlights Marshal, an instrumentation SDK, enables runtime inspection of VCL and FMX applications via inspectors that are assigned by class type. Marshal provides navigation of your applications forms, datamodules and components. Marshal is built with the FNC UI Pack. Explore how easy it is to inspect your application with Marshal.  Explore your components with Object Plus, an integration of the FNC Object Inspector that captures the detail of your changes along with before and after screen shots. TDatasets get an extended analysis by attaching an FNC Grid with an export to csv. FireDAC connections get the ability to run adhoc SQL. Caddie is the offline analysis and organization component of RunTime ToolKit. Inspection generates a lot of data and Caddie gives you the ability to review a runtime session at a later date. Additional image analysis is executed as part of session collection to provide more visual clues on before and after images. Summary Deputy connects the circle of processes that RunTime ToolKit is tailored to provide for you as the developer. Making your tasks easier via inspection, data collection and analysis is the focus of RunTime ToolKit. Learn more here  Request for Feedback Your comments are essential for improving these tools and I am listening for feedback.  Please leave a comment below, contact me via email ( web@swiftexpat.com ) , open an issue on GitHub (https://github.com/SwiftExpat/Deputy ), or reach out on Delphi Praxis. Neil Laskowski 

Read More

10 Signs You Work With The Best JS Framework

Sencha

Nowadays, most web developers prefer to use frameworks for their developments as they provide a more reliable and efficient way to build modern JavaScript-based web applications. However, it will be best if you can find the most suitable and user-friendly framework before starting. Sencha Ext JS is one of the best JS frameworks you can find on the internet. In this post, let’s discuss ten signs of a good JS framework. Is It Easy to Set Up and Use? The framework you select should be easy to set up and use. Another essential fact is that the framework should be cross-platform compatible as you have to share your code across multiple devices and developers who use different operating systems. Furthermore, if the framework has documentation, it is easy for developers to go through and refer to when they struggle with something or need to find out new features. Sencha Ext JS is easy to set up and use. This framework is appropriate for JavaScript-based development as it supports numerous popular libraries such as Angular, React, and Next.js. Does Your JS Framework Auto-Generate Code? Whatever the JS library you use, you have to implement different components using HTML5. In that case, developers have to put some effort into constructing HTML forms, components such as login, registration, dashboards, and contact pages. However, auto-generating code instead of coding manually will save time in the development phase and reduce human errors. Sencha provides that feature for busy developers. As the best JS framework, Sencha comes with a visual app builder called Sencha Architect. It lets you generate excellent user interfaces by dragging and dropping different UI components. How do I Debug the JS Code? Developers have to put considerable effort into debugging during the development and maintenance phases of SDLC. The reason is that you have to analyze the code step by step once you overcome an incident or issue. That process will be fast and effective if the JS framework you use supports debugging. Sencha Inspector provides debugging and troubleshooting features in the ExtJS framework. It equips smart debugging features, such as direct access to classes, objects, and components. Thus, you can analyze the JS code for issues and identify problems such as over nesting components and the number of layouts required to operate the JS app in optimal performance. Furthermore, you can use Sencha Inspector to perform unit testing and end-to-end testing of apps built using ExtAngular, Angular, and ExtReact on different operating systems and cross-browser platforms. Can I Share Code Snippets? You will have to share code snippets with your colleagues and other developers during the implementation of your JavaScript application. So developers use different techniques to share code, such as sharing code via GitHub or GitLab and pair programming. Now you can easily create & share code using Sencha Fiddle, which is an online platform. You don’t have to set up anything locally. Yet it brings you the look and feels of a local development environment. It also supports all the browsers and works on any operating system. You have to log in with Sencha forum user credentials and share the code using the fiddle URL to share code snippets. Can You Use GraphQL with Your JS Framework? Today most developers use GraphQL to control the data required through an […]

Read More

Abonează-te la noutăți

Primiți noutăți și oferte despre SDLC tools

Close Menu