[RFC]: adding support for visualizing benchmark results

[Vinit-Pandit]

Full name

Pandit Vinit

University status

Yes

University name

Parul University

University program

B-tech in CSE

Expected graduation

2025 SEP

Short biography

I'am forth year undergraduate student at the Parul university of Vadodara Gujarat. I am always starve to learning new things and gain deep understanding of the core concepts related to software, and I really love to deep dive into topics I'm unfamiliar with, love to understand the intuation behind maths concepts.

Currently, I’ve just completed an internship with a company called Slash Mark. I also previously finished a summer internship with Bytxl, where I learned about the MVC architecture and how to build scalable backends with Node.

I have worked with JavaScript, React, Node.js, and C++. I used JavaScript to build a Domain specific language for manipulating JSON data, and React Native to develop Android apps.

Timezone

Indian Standard Time

Contact details

[email protected]

Platform

Linux

Editor

I often toggle between NeoVim and VS Code. For working on Stdlib, I use VS Code because it's user-friendly and works well with remote containers. However, for smaller tasks, I strongly prefer and recommend using ⚡ speed NeoVim.

Programming experience

I’ve worked on a lot of projects in the past—such as Android apps, websites, and a DSL I created—but looking back, many of them now feel pretty outdated and straightforward. Over the last six months, I’ve gained valuable experience that helped me realize software engineering isn’t just about building a basic frontend or backend. It’s more about writing well-designed, readable, and well-documented code and most important reading the others code.

• JSON DSL A domain-specific language built to transform JSON data based on defined transformation rules.

• Pu_mato client side
  server side A food delivery Android app built for a college project, designed to order food from nearby food services

• Connectify A straightforward chat app built using Socket.IO to explore and understand WebSockets .

JavaScript experience

I’ve gained solid experience with JavaScript, thanks in part to the book You Don’t Know JS Yet.

I have build the web application, android application using react.

Sometimes, dynamic typing cause overhead during debugging, which is why I prefer TypeScript over plain JavaScript. :) .

Node.js experience

I mostly enjoy working on the backend. I have used Node.js to build APIs following the MVC architecture, developed a web servers with features like load balancing, and played concepts like microservices and orchestration.

C/Fortran experience

I have a good understanding of C. It's one of the most fundamental languages that provides deep insights into low-level programming concepts. I’ve explored and worked with key concepts like memory management, pointers, structs, and more.

Currently, I don’t have any experience with Fortran.

Interest in stdlib

As a beginner in software development, exploring the stdlib project has been a turning point for me. What really fascinated me was its architecture, the strict code styling guidelines, and the extensive use of tools and automation. The way almost every part of the workflow from testing to documentation is automated taught me how large-scale codebases are maintained efficiently. It also helped me realize the importance of clean code, readability, consistency, and well-written documentation.

Beyond the development practices, stdlib gave me my first real introduction to the world of numerical computing.
Working with the library sparked my interest in how complex mathematical operations are implemented under the hood, and motivated me to explore topics like BLAS, LAPACK, and precision computing.

"If I’m being honest, I love understanding mathematical algorithms but they require a strong background and solid qualifications in math. So, what really interests me are REPLs and automation tools like linting rules, automation bots for PRs, CI/CD pipelines, GitHub workflows, and remote container support.

Version control

Yes

Contributions to stdlib

I have multiple Prs on stdlib. The work which give me the best experiences are

• add eager evaluation of side-effect free code
• js implementation of blas/bas/cdotc
• currently working add base interface of Unicode plot and add implementation of bar plot

Merge Pull Requests

Merged PRs

Open Pull Requests

Open Prs

stdlib showcase

Nothing to show here.

Goals

My goal with the project adding support for visualizing benchmark results is to provide support to visulize the benchmark results .

Goals

• Build a TAP parser to convert raw benchmark results into structured data facets.
• Develop a Unicode/ANSI plotting package to visualize benchmark data by feeding in the parsed facets.

Details

Unicode / Ansi plot

• To visualize benchmark results effectively, we need a plotting package that can process data facets and render customizable plots in the terminal.

• My approach is to first build a base plotting package, which acts as a foundation for all visualizations. This package will expose an interface for other plot types and support features such as:

  • Adding glyphs ,
  • Axis labels and
  • Custom colors at specific positions within a 2D array that stores glyphs and color data for terminal rendering

• A bar plot or grouped bar plot is ideal for visualizing benchmark results. So the next step is to build a dedicated bar plot package

• I’ve already made progress to build the base package and bar plot

Code

benchmark2.webm

In this example, I manually created data facets from benchmark output and used them for continuously visualizing rate vs size plot for benchmark result of @stdlib/blas/ddot.

• A state manager function is used to manage the options for the bar plot, which are continuously updated as new data facets are received.
• The script accepts arguments to determine which params and scopes go on which axes
• It also supports a filter object to narrow down the data facets to only those that are relevant.
• In production, this state manager will be replaced by an interface that filters and structures the data facets, pushing them directly to the bar chart package.

Parser

Since there are clear references available for building the Tap Parser, it can be relatively straightforward to implement. The parser can handle various types of objects such as test, asserts , plan something similar to https://github.com/tap-format/parser.

Since we use a special DSL for benchmark descriptions in the format ::namespace:method:paramsto include metadata, we need to add utility functions to @stdlib/bench/*. The goal is to provide a more structured way to embed and parse benchmark metadata within descriptions. This metadata is crucial for accurate visualization of the benchmarks.

I will provide more detail on parser here

Challenges

• Filtering desired data facets: Benchmarks can include multiple scopes and parameters. It’s important to ensure the correct facets are selected and validated before plotting.

• Dynamic plot generation based on user input: Since there can be many combinations of scopes and parameters, automatically generating all possible visualizations would be chaotic. Instead, we need to:

  • Provide default visualizations
  • Offer a command-based interface for generating plots based on selected parameters

• Visualizing across different scopes: Parameters across scopes may not match. For example:

  • In blas/base/ddot, there are two scopes:
    1. stacks with parameters size, ndims, and shape
    2. vectors with the parameter len
  • Since these scopes have different parameters, it becomes hard to compare or merge their results directly. One solution is to average out or normalize results across parameters, but this requires careful consideration.

Additional supportive goals

• CI / CD for the benchmark visulization
• More unicode plots like line chart

These additional goals are intended to be pursued independently of the main project, even after the GSoC timeline.

Why this project?

There are several projects that interest me, such as “Reimagine the stdlib plot API and implementation” and “Extend stdlib's doctesting approach to C examples.”
However, I didn’t choose the first one due to its lower priority, and while I was interested in the second, I couldn’t commit to it because of a tight schedule over the past two months. Additionally, that project was proposed a bit late, which made it harder for me to take action in time.

In short, I chose a unique project one that allows me to learn a lot during the process while also contributing meaningfully to the future development of stdlib.

Qualifications

• I already have strong progress in base package and bar plot which explain my idea .

• I also have multiple PRs in stdlib, which has made me familiar with its code style and architecture.

• Apart from it i have completed the online Data Analytics course

Prior art

There are many prior arts are avialble for the the building the parsers one of them which interest me is

• https://github.com/tap-format/parser

For unicode plot

• https://github.com/piccolomo/plotext
• https://github.com/JuliaPlots/UnicodePlots.jl

Commitment

I am fully available for this project and ready to provide 350 hours or more, with a commitment of 30 to 45 hours per week. I will maintain steady progress with regular pull requests and i am dedicated to staying responsive and engaged through Gitter and GitHub.

I’ve already developed a some working prototype in my forked repository, which has helped me gain confidence and clarity on the implementation. This early progress has provided a strong foundation to move forward efficiently and deliver meaningful results.

In addition to technical contribution, I value clear communication, collaboration, and proactive problem-solving. I’m always open to feedback and eager to align my work with the team’s goals and vision. You can count on me to be consistent, reliable, and fully invested in the success of this project.

Schedule

Assuming a 12 week schedule,

• Community Bonding Period:

During this period, I will clarify the mentors' goals and expectations for the project. I will explore areas of the project I haven't fully understood yet, such as the CI/CD architecture, which will help me design and implement CI/CD pipelines for this project.
Additionally I will begin researching the implementation of a TAP parser and perform a deep dive into the Tap specifications to fully understand the parser's requirements.

• Week 1 - Week 2: Implement Tap Parser

  • Study the Tap 14 specification.
  • Develop the parser to output the proper formatted benchmark result.
  • Write the test cases to test the parser functionality.

• Week 3: Week 4: **Implement Unicode Base Plot **

  • Study the general requirement of the all plots.
  • Build the base plot package.
  • Ensure the base plot is flexible enough to support other Unicode-based plots.
  • Write the test cases to ensure functionality and extensibility.

• Week 5: Week 6: (midterm) Feedback and Review and Bar Plot Initialization

  • Collect feedback from mentors on the TAP parser and base plot.
  • Refactor and make changes based on feedback.
  • Gather requirement for the bar plot.
  • Begin implementing the bar plot.

• Week 7: -Week 8: Completion of bar plot

  • Finalize the implementation of the bar plot.
  • Add the tests for bar plot.
  • Integrate the parser with bar plot to visualize benchmark data.
  • Collaborate with mentors to define CI / CD requirement.
  • Begin the Implementation of CI CD checks.

• Week 9 - Week 10 : Completion of CI / CD checks

  • Implement CI/CD checks for benchmark visualization workflows.
  • Write comprehensive documentation for the TAP parser, base plot, and bar plot.

• Week 11 Week 12: Wrap up

  • Implement any final suggestions or additional features requested by mentors.
  • Verify that all project requirements are met.
  • Perform final polishing and cleanup of code and documentation.

• Final Week: Submission and Future planning

  • Submit the final version of the project with all deliverables.
  • Discuss possible future improvements and feature additions with mentors.
  • Share lessons learned and receive final feedback to reflect on the development process.

Thank you very much to read my proposal !

Related issues

Gsoc
#2010

Checklist

• I have read and understood the Code of Conduct.
• I have read and understood the application materials found in this repository.
• I understand that plagiarism will not be tolerated, and I have authored this application in my own words.
• I have read and understood the patch requirement which is necessary for my application to be considered for acceptance.
• I have read and understood the stdlib showcase requirement which is necessary for my application to be considered for acceptance.
• The issue name begins with [RFC]: and succinctly describes your proposal.
• I understand that, in order to apply to be a GSoC contributor, I must submit my final application to https://summerofcode.withgoogle.com/ before the submission deadline.

[Vinit-Pandit]

@kgryte @Planeshifter @gunjjoshi
Apologies for the delayed proposal 😓. The past month has been filled with unexpected events and a heavy load of college work😭.
Looking forward to your feedback 😃.
Thank you!

[kgryte]

@Vinit-Pandit Thank you for opening this RFC. A few comments/questions:

1. It would be good to include the development of utilities for parsing benchmark descriptions. We currently use an undocumented DSL (e.g., ::namespace:method:params). For faceting, it would be good to actually codify this DSL and provide parsers. These can be added to @stdlib/bench/*.
2. Similar to other TAP utilities, the benchmark visualizer should provide a CLI which reads stdin, performs parsing, and then outputs the viz. This fits in the general mold of "tap reporters", several of which we have as dev deps in the codebase (e.g., tap-min, etc). May be worthwhile taking a peak at those packages for inspiration.
3. In your demo, you showed the benchmark plots update in real-time. An alternative approach is to simply read in an entire TAP stream, perform faceting, and then show a single faceted plot. While real-time Unicode plots has use cases, I don't think having real-time updating benchmark result plots is critical, and it will likely simplify things and be more performant if you first collect all benchmark results, analyze them, and then plot.
4. When we run benchmarks for a package, this may result in multiple benchmark files being executed, with separate TAP results. It would be nice to have some way of combining results from multiple files.