Writing a Shell in 25 Lines of Ruby Code

If you use Linux or Mac, every time you open a terminal you are using a shell application. A shell is just an interface that helps you execute commands in your system.

In addition to that, the shell also hosts environment variables & has useful features like a command history and auto-completion.

If you are the kind of person that likes to learn how things work under the hood, this post will be perfect for you!

How Does a Shell Work?

To build our own shell application let’s think about what a shell really is: first, there is a prompt, usually with some extra information like your current user & current directory, then you type a command & when you press enter the results are displayed on your screen.

Yeah, that sounds pretty basic, but doesn’t this remind you of something?

If you are thinking of pry then you are right! A shell in basically a REPL (Read-Eval-Print-Loop) for your operating system.

So knowing that we can write our first version of your shell:

This will give us a minimal, but functional shell. We can improve this by using a library that many other REPL-like applications use. That library is called Readline.

Using The Readline Library

Readline is part of the Ruby Standard Library, so there is nothing to install, you just need to require it.

One of the advantages of using Readline is that it can keep a command history automatically for us. It can also take care of printing the command prompt & many other things.

Here is v2 of our shell, this time using Readline:

This is great, we got rid of the two puts for the prompt & now we have access to some powerful capabilities from Readline. For example, we can use keyboard shortcuts to delete a word (CTRL + W) or even search the history (CTRL + R)!

Let’s add a new command to print the full history:

Fun fact: If you try this code in pry you will get pry’s command history! The reason is that pry is also using Readline, and Readline::HISTORY is shared state.

Now you can type hist to get your command history

Adding Auto-Completion

Thanks to the auto-completion feature of your favorite shell you will be able to save a lot of typing. Readline makes it really easy to integrate this feature into your shell.

Let’s start by auto-completing commands from our history.

Example:

With this code you should be able to auto-complete previously typed commands by pressing the <tab> key. Now let’s take this a step further & add directory auto-completion.

Example:

The completion_proc returns the list of possible candidates, in this case we just need to check if the typed string is part of a directory name by using Dir.glob. Readline will take care of the rest!

Implementing The System Method

Now you should have a working shell, with history & auto-completion, not too bad for 25 lines of code

But there is something that I want to dig deeper into, so you can get some insights on what is going on behind the scenes of actually executing a command.

This is done by the system method, in C this method just sends your command to /bin/sh, which is a shell application. Let’s see how you can implement what /bin/sh does in Ruby.

Note: This will only work on Linux / Mac

The system method:

What happens here is that fork creates a new copy of the current process, then this process is replaced by the command we want to run via the exec method. This is a very common pattern in Linux programming.

If you don’t fork then the current process is replaced, which means that when the command you are running (ls, cd or anything else) is done then your Ruby program will terminate with it.

You can see that happening here:

Conclusion

In this post you learned that a shell is a REPL-like interface (think irb / pry) for interacting with your system. You also learned how to build your own shell by using the powerful Readline library, which provides many built-in features like history & auto-completion (but you have to define how that works).

And after that you learned about the fork + exec pattern commonly used in Linux programming projects.

If you enjoyed this post could you do me a favor & share it with all your Ruby friends? It will help the blog grow & more people will be able to learn

Building Your Own Linux Tools with Ruby: A Practical Guide

Tools like ps, top & netstat are great, they give you a lot of information about what’s going on your system.

But how do they work? Where do they get all their information from?

In this post we will recreate three popular Linux tools together. You are going to get a 2×1 meal, learn Ruby & Linux at the same time!

Finding Status Information

So let’s try answering the question of where all these tools find their info. The answer is in the /proc filesystem.

If you look inside the /proc directory it will look like a bunch of directories & files, just like any other directory on your computer. But the thing is that these aren’t real files, it’s just a way for the Linux kernel to expose data to users.

It’s very convenient because they can be treated like normal files, which means that you can read them without any special tools. In the Linux world a lot of things work like this, if you want to see another example take a look at the /dev directory.

Now that we understand what we are dealing with, let’s take a look at the contents of the /proc directory…

This is just a small sample, but you can quickly notice a pattern. What are all those numbers? Well, it turns out these are PIDs (Process IDs). Every entry contains info about a specific process.

If you run ps you can see how every process has a PID associated with it:

From this we can deduce that what ps does is just iterate over the /proc directory & print the info it finds.

Let’s see what is inside one of those numbered directories:

That’s just a sample to save space, but I encourage you to take a look at the full list.

Here are some important / interesting entries:

Entry	Description
comm	Name of the program
cmdline	Command used to launch this process
environ	Environment variables that this process was started with
status	Process status (running, sleeping…) & memory usage
fd	Directory that contains file descriptors (open files, sockets…)

Now that we know this we should be able to start writing some tools!

Process Listing

Let’s start by just getting a list of all the directories under /proc. We can do this using the Dir class.

Example:

Notice how I used a number range, the reason is that there are other files under /proc that we don’t care about right now, we only want the numbered directories.

Now we can iterate over this list and print two columns, one with the PID & another with the program name.

Example:

And this is the output:

Hey, it looks like we just made ps! Yeah, it doesn’t support all the fancy options from the original, but we made something work.

Who Is Listening?

Let’s try to replicate netstat now, this is what the output looks like (with -ant as flags).

Where can we find this information? If you said “inside /proc” you’re right! To be more specific you can find it in /proc/net/tcp.

But there is a little problem, this doesn’t look anything like the netstat output!

What this means is that we need to do some parsing with regular expressions. For now let’s just worry about the local address & the status.

Here is the regex I came up with:

This will give us some hexadecimal values that we need to convert into decimal. Let’s create a class that will do this for us.

The only thing left is to print the results in a pretty table format.

Example output:

Yes, this gem is awesome!

I just found about it & looks like I won’t have to fumble around with ljust / rjust again

Stop Using My Port!

Have you ever seen this message?

Umm… I wonder what is using that port…

Ah, so there is our culprit! Now we can stop this program if we don’t want it to be running & that will free our port. How did the fuser program find out who was using this port?

You guessed it! The /proc filesystem again.

In fact, it combines two things we have covered already: walking through the process list & reading active connections from /proc/net/tcp.

We just need one extra step:
Find a way to match the open port info with the PID.

If we look at the TCP data that we can get from /proc/net/tcp, the PID is not there. But we can use the inode number.

“An inode is a data structure used to represent a filesystem object.” – Wikipedia

How can we use the inode to find the matching process? If we look under the fd directory of a process that we know has an open port, we will find a line like this:

The number between brackets is the inode number. So now all we have to do is iterate over all the files & we will find the matching process.

Here is one way to do that:

Example output:

Please note that you will need to run this code as root or as the process owner. Otherwise you won’t be able to read the process details inside /proc.

Conclusion

In this post you learned that Linux exposes a lot of data via the virtual /proc filesystem. You also learned how to recreate popular Linux tools like ps, netstat & fuser by using the data under /proc.

Don’t forget to subscribe to the newsletter below so you don’t miss the next post (and get some free gifts I prepared for you)

Ruby Ranges: How Do They Work?

Have you ever wondered how ranges work in Ruby?

Even if you haven’t, isn’t it fun to discover how things work under the hood?

That’s exactly what I’m going to show you in this post.

Understanding Ranges

Just as a reminder, this is what a Ruby range looks like:

The parenthesis are not necessary to define a Range, but if you want to call methods on your range you will need them (otherwise you are calling the method on the 2nd element of the range, instead of the range itself).

The Range class includes Enumerable, so you get all the powerful iteration methods without having to convert the range into an array.

Range has some useful methods, like the step method.

Example:

Other Range methods to be aware of are cover? & include?. It would be a mistake to think that they do the same thing, because they don’t.

The include? method just does what you would expect, check for inclusion inside the range. So it would be equivalent to expanding the Range into an Array and checking if something is in there.

But cover? is different, all it does is check against the initial & ending values of the range (begin <= obj <= end), which can yield unexpected results.

Example:

The cover? example is equivalent to:

The reason this returns true is that strings are compared character by character. Since “a” comes before “c”, the characters that come after the “c” don’t matter.

Range Implementation

Ranges are not limited to numbers & letters, you can use any objects as long as they implement the following methods: <=> and succ.

For example, here is a time range:

So how does this work? Let’s take a look at this implementation:

I added some comments to help you understand what is going on. The idea is that we keep calling the next method on the first object until it is equal to the second one, the assumption is that they will eventually meet.

Custom Class Ranges

Most of the time you will be using number & character ranges, but it’s still good to know how you can use ranges in a custom class.

Example:

The key here is to make sure that you implement the <=> & succ methods correctly. If you want to use the include? method you need to include the Comparable module, which adds methods like ==, <, and > (all based on the results of the <=> method).

Conclusion

In this article you have learned how ranges work in Ruby so you can understand them better & implement your own objects that support range operations.

Don’t forget to subscribe to the newsletter below so you don’t miss the next post

5 Useful Examples From The Ruby Standard Library

The Ruby Standard Library is a series of modules & classes that come with Ruby but are not part of the language itself.

These classes offer a variety of utilities like Base64 encoding, prime number generation & DNS resolution.

In this article I’m going to show you 5 of these classes with useful examples.

Unique Items with Set

Set is a data structure that guarantees unique items.

Here is an example:

The implementation of this data structure is based on the Hash class, so there is no Array-like indexing.

But if you just need a bucket of data where all elements need to be unique all the time, then the Set might be what you are looking for.

Logging Messages

If you need to log some error / debug message Ruby has you covered with the Logger class. This class provides everything you need to start logging.

In fact, this class is what Rails uses by default.

To use the Logger class you can simply create a Logger object & give it an output stream (or a file name) as a parameter. Then you can use the different logging levels to register your message.

The logging levels are: debug, info, warn, error & fatal.

Example:

This produces the following output:

The first character is an abbreviated form of the logging level (I for Info, W for Warn)… Then you have a timestamp, and the current process id (which you can get in Ruby using Process.pid).

Finally, you have the full logging level & the actual message. You can change this format by providing a new formatter.

Working with Prime Numbers

Maybe you don’t need to deal with prime numbers on a day-to-day basis, but it’s still good to know (specially if you like programming challenges) that Ruby has good support for them via the Prime class.

When you require Prime it will add the prime? method to Fixnum.

This class also includes a prime number generator.

Example:

Using StringIO

The StringIO class allows you to create a string that behaves like an IO object. This means that you can work with this string like if you were reading from a file or STDIN (Standard Input).

Here is an example:

Notice a few things: when you add data into your StringIO object it will not add spaces or newlines for you, and to get the actual string you need to call the string method. Also you can’t use array indexing to access individual characters, like you can with a regular string.

When is this useful? Well, sometimes you may want to substitute a file or some other IO object for another object that you have more control over. For example, in a testing environment you can replace STDOUT with a StringIO object.

You can see a real world example from Rails here: https://github.com/rails/rails/blob/52ce6ece8c8f74064bb64e0a0b1ddd83092718e1/activesupport/test/logger_test.rb

Working with Paths

The Pathname class wraps several file-system exploring utilities, like Dir & File, in a much more powerful class.

While the method names are the same, they return Pathname objects instead of strings or arrays. And what this means is that you can keep working with the results with all the file-related methods.

This is a great example:

If you tried to do this with Dir, you would have to use this code, which is not as elegant.

Give this one a try the next time you need to do a file-system related task

Conclusion

I hope you found these examples useful! Make sure to explore the Standard Library a bit more so you can learn what it can do for you.

Don’t forget to share this post if you liked it

How to Generate Weighted Random Numbers

Random numbers usually follow what we call a ‘uniform distribution’, meaning that there is the same chance that any of the numbers is picked.

But if you want some numbers to be picked more often than others you will need a different strategy: a weighted random number generator.

Some practical applications include:

• the loot table in a video game, where enemies can drop different items with varying drop rates.
• a raffle, where people with more tickets have more chances to win.

Simple Strategy

If you think about the raffle example you can come up with an obvious solution: generate an array which has one copy of the item for each ‘ticket’.

For example, if John buys 4 raffle tickets and David only buys 1, John will have 4 times more chances to win than David.

Here is a working implementation:

I’m adding the person’s name once for every ticket they bought, and then I pick a random name from that list. By virtue of being on the list more times, it will increase the chances of that name being picked.

I like this approach because it’s very simple and once you have your list it’s very fast to pick a winner.

Sum of Weights

There is another way you can do this that is more memory-efficient, the trade-off is that picking a random value is slower.

The idea is to pick a random number between 1 and the sum of all the weights, and then loop until you find a weight that is lower or equal than this number.

Here is the code:

This code takes in a hash where the keys are the items and the values are the weights. You can call this method like this:

You can test if this works as expected by looking at the distribution of the results after running it many times.

Here is an example:

Run this a few times and look at the output to see if the ratio is what it should be.

Conclusion

While there are more sophisticated algorithms, these two should serve you well. I hope you found this article useful, please share it with your friends so I can keep writing more!