A practical example of RakuAST

Mai 26, às 10:01


6 min de leitura


0 leituras

A while ago someone asked on #raku if it is possible to create a Raku character class with the valid characters being supplied by a string. This is not possible in Raku at the moment. But it is possible using...
A practical example of RakuAST

A while ago someone asked on #raku if it is possible to create a Raku character class with the valid characters being supplied by a string. This is not possible in Raku at the moment. But it is possible using RakuAST!

Let's first see how one can create characters classes with RakuAST by applying the .AST method to an example.

By the way, all of these examples assume that a use experimental :rakuast; or use v6.e.PREVIEW is active.

say 'my token {<[abc]>}'.AST.statements.head.expression;

Enter fullscreen mode Exit fullscreen mode

What this does is

  • create the AST (.AST) for an anonymous token with a charclass for the letters "a", "b" and "c"
  • then skips to the first statement (.statements.head)
  • then skips to its .expression

because the .AST returns a statement list, and we're only interested in the expression of the first statement.

The result is:
 body =>
 elements => ("a"),"b"),"c"),

Enter fullscreen mode Exit fullscreen mode

So it looks like each character in the charclass is a separate RakuAST::Regex::CharClassEnumerationElement::Character object. With this knowledge, it is pretty easy to make a custom token for the characters in a given string. Let's create a subroutine "chars-matcher" that will create a token with a charclass of the characters for a given string:

sub chars-matcher($string) {
 my @elements = $ {$_)
 body =>

Enter fullscreen mode Exit fullscreen mode

First we create an array "@elements" and fill that with an enumeration object for each unique char in the given string ($string.comb.unique). And then we create the TokenDeclaration object as from the example, but with the @elements array as the specification of the characters. And then we convert that into an actual usable token by running .EVAL on it.

An example of its usage:

my $matcher = chars-matcher("Anna Mae Bullock");
say "Tina Turner" ~~ $matcher; # 「n」
say "Tina Turner" ~~ / $matcher+ /; # 「na 」

Enter fullscreen mode Exit fullscreen mode

As you can see, you can use the generated token direct in a smart-match. Or you can use it as part of a more complicated regex.

If you're ready to further dabble with RakuAST, it is probably a good idea to know a little bit of how it is currently being implemented. So let's dive a little bit into that, to better understand some of the errors you might encounter when writing Raku Programming Language code to create ASTs.

Prerequisites for RakuAST

It is the intent to have all Raku source code be parsed by the (new) Raku grammar (and associated Actions) in the future, just as it is now with the legacy grammar. Since the Raku core setting (which contains the Raku code for most of the implementation of the Raku Programming Language) must also be parsed by this, it implies that the RakuAST classes must exist before there is any Raku Programming Language.

This is a chicken-and-egg problem that is solved in Rakudo by the so-called "bootstrap". This is quite a sizeable chunk of NQP code that "manually" creates enough functionality to allow the Raku core setting to build itself up to a fully functional implementation of the Raku Programming Language.

When Jonathan Worthington started the RakuAST project, they didn't want to have to implement all of that functionality in NQP yet again. So they devised a neat hack by creating a rather simple parser that would read Raku-like code, and convert that to NQP source code that would create all of the 360+ RakuAST classes when run. Of course, that Raku-like code still does not have the full Raku capabilities, but it does make the implementation task a lot easier than it would have been if it had be all written in NQP.

Example of a RakuAST class' internals

Let's have look a simple example of such a RakuAST class. For instance, the RakuAST::StrLiteral class that we've seen earlier in the first instalment.

class RakuAST::StrLiteral is RakuAST::Literal {
 has Str $.value;
 method new(Str $value) {
 my $obj := nqp::create(self);
 nqp::bindattr($obj, RakuAST::StrLiteral, '$!value', $value);
 # other methods
 method IMPL-EXPR-QAST(RakuAST::IMPL::QASTContext $context) {
 my $value := $!value;
 my $wval := :$value );$wval,'Ss',$value))))

Enter fullscreen mode Exit fullscreen mode

For simplicity's sake, only two methods are shown. The new method, taking a single positional Str $value. Which needs some NQP to create the object and bind the value to the $!value attribute.

And we see an IMPL-EXPR-QAST method. That method will be called whenever that RakuAST object needs to generate QAST (a precursor to actual bytecode) for itself.

If you find this very interesting, you probably want to read the RakuAST README. And the actual source code of the RakuAST classes can be found in the same directory. And if you're really feeling adventurous and you have the Rakudo repository checked out, you can have a look at the generated NQP code in gen/moar/ast.nqp.

What's the point?

So why am I even mentioning this? Because the RakuAST classes look like they're actual Raku classes, but they're really NQP subroutines wrapped up to appear like Raku classes. Which results in unexpected failure modes if there's some error in your calls to RakuAST classes. In other words: the edges are a little bit sharper with RakuAST classes, and LTA error messages can and will happen. It's one of the "benefits" of living on the edge!

Of course, as a user of RakuAST classes, you should only be interested in the new method, and any other non-internal methods. Sadly, it is way too early in the bootstrap to mark internal methods with an is implementation-detail trait, so another heuristic is needed. And that would be: "consider any ALL-CAPS methods to be off-limits".


This installment gives an actual example of how you can use RakuAST in your code today. And gives some technical background on the implementation of RakuAST classes, which still is a little sharp around the edges.

The intended audience are those people willing to be early adopters of these exciting new features in the Raku Programming Language.

Continue lendo

Tech Crunch

Deal Dive: Why this startup chose to sell itself over raising a Series A
Heroes Jobs was recently acquired after realizing its company was better off not alone Not all startups are built for a billion-dollar exit — or to grow as a stand-alone company at all. The...

Hoje, às 16:00


Mobile App ideas
What interesting, useful and simple mobile apps can be made? What do you think? Maybe there are problems that can be solved with this app? Share your ideas. Of course, todo list and weather apps are a great...

Hoje, às 14:51


The Evolution and Impact of MLOps: understanding MLOps
Machine Learning Operations, or MLOps, is the missing bridge between machine learning, data science, and data engineering. It has emerged as the link that unifies these functions more seamlessly than ever...

Hoje, às 14:00


117 Stories To Learn About Programming Tips | HackerNoon
May 27th 2023 New Story17m by @learn Too Long; Didn't ReadPeople Mentionedscience#programming-tips#learn#[email protected] LearnReceive Stories from @learnGet free API security automated scan in...

Hoje, às 13:56


279 Stories To Learn About Programming | HackerNoon
May 27th 2023 New Story51m by @learn Too Long; Didn't ReadPeople Mentionedprogramming#programming#learn#[email protected] LearnReceive Stories from @learnGet free API security automated scan in...

Hoje, às 13:56


How to use Mapbox in Next.js
Table of content Mapbox Setting up the Mapbox key Next.js Dependencies Adding key to Next.js configuration Initializing the map Adding styles Adding the map controls Loading the data Adding the...

Hoje, às 13:15


Add MUI v5 Theme Switcher in NextJs/React
Hello guys today i will show you how to add MUI Theme Switcher in your React App. Prerequisite of this tutorial is to have intermediate knowledge of React and MUI (formally known as material-UI) Lets...

Hoje, às 13:03


50 filmes que todo empreendedor deve assistir
Índice Fyre Festival – Fiasco no Caribe (2019)Como Ser Warren Buffett (2017)Fome de Poder (2016)Cães de Guerra (2016)Generation Startup (2016)Steve Jobs (2015)Joy: O Nome do Sucesso (2015)A Grande Aposta...

Hoje, às 13:00


Binary tree
What is binary tree and when we could use them? A binary tree is a fundamental tree data structure that consists of nodes connected in a hierarchical manner. Each node in a binary tree can have at most two...

Hoje, às 12:47


Startups of The Year 2023: 200+ Startups Nominated in Seattle | HackerNoon
Start WritingNotificationssee moreStartups of The Year 2023: 200+ Startups Nominated in Seattle [email protected] 27th 2023 New Story3m by @startups Too Long; Didn't ReadHackerNoon, the technology publishing...

Hoje, às 12:18