Black Bytes
Share this post!

All posts by Jesus Castello

linux tools ruby

Building Your Own Linux Tools with Ruby: A Practical Guide

Tools like ps, top & netstat are great, they give you a lot of information about what’s going with your system.

But how do they work? Where do they get all their information from?

In this post we will recreate three popular Linux tools together. You are going to get a 2×1 meal, learn Ruby & Linux at the same time! πŸ™‚

Finding Status Information

So let’s try answering the question of where all these tools find their info. The answer is in the /proc filesystem.

If you look inside the /proc directory it will look like a bunch of directories & files, just like any other directory on your computer. But the thing is that these aren’t real files, it’s just a way for the Linux kernel to expose data to users.

It’s very convenient because they can be treated like normal files, which means that you can read them without any special tools. In the Linux world a lot of things work like this, if you want to see another example take a look at the /dev directory.

Now that we understand what we are dealing with, let’s take a look at the contents of the /proc directory…

This is just a small sample, but you can quickly notice a pattern. What are all those numbers? Well, it turns out these are PIDs (Process IDs). Every entry contains info about a specific process.

If you run ps you can see how every process has a PID associated with it:

From this we can deduce that what ps does is just iterate over the /proc directory & print the info it finds.

Let’s see what is inside one of those numbered directories:

That’s just a sample to save space, but I encourage you to take a look at the full list.

Here are some important / interesting entries:

Entry Description
comm Name of the program
cmdline Command used to launch this process
environ Environment variables that this process was started with
status Process status (running, sleeping…) & memory usage
fd Directory that contains file descriptors (open files, sockets…)

Now that we know this we should be able to start writing some tools!

Process Listing

Let’s start by just getting a list of all the directories under /proc. We can do this using the Dir class.

Example:

Notice how I used a number range, the reason is that there are other files under /proc that we don’t care about right now, we only want the numbered directories.

Now we can iterate over this list and print two columns, one with the PID & another with the program name.

Example:

And this is the output:

Hey, it looks like we just made ps! Yeah, it doesn’t support all the fancy options from the original, but we made something work.

Who Is Listening?

Let’s try to replicate netstat now, this is what the output looks like (with -ant as flags).

Where can we find this information? If you said “inside /proc” you’re right! To be more specific you can find it in /proc/net/tcp.

But there is a little problem, this doesn’t look anything like the netstat output!

What this means is that we need to do some parsing with regular expressions. For now let’s just worry about the local address & the status.

Here is the regex I came up with:

This will give us some hexadecimal values that we need to convert into decimal. Let’s create a class that will do this for us.

The only thing left is to print the results in a pretty table format.

Example output:

Yes, this gem is awesome!

I just found about it & looks like I won’t have to fumble around with ljust / rjust again πŸ™‚

Stop Using My Port!

Have you ever seen this message?

Umm… I wonder what is using that port…

Ah, so there is our culprit! Now we can stop this program if we don’t want it to be running & that will free our port. How did the fuser program find out who was using this port?

You guessed it! The /proc filesystem again.

In fact, it combines two things we have covered already: walking through the process list & reading active connections from /proc/net/tcp.

We just need one extra step:
Find a way to match the open port info with the PID.

If we look at the TCP data that we can get from /proc/net/tcp, the PID is not there. But we can use the inode number.

“An inode is a data structure used to represent a filesystem object.” – Wikipedia

How can we use the inode to find the matching process? If we look under the fd directory of a process that we know has an open port, we will find a line like this:

The number between brackets is the inode number. So now all we have to do is iterate over all the files & we will find the matching process.

Here is one way to do that:

Example output:

Please note that you will need to run this code as root or as the process owner. Otherwise you won’t be able to read the process details inside /proc.

Conclusion

In this post you learned that Linux exposes a lot of data via the virtual /proc filesystem. You also learned how to recreate popular Linux tools like ps, netstat & fuser by using the data under /proc.

Don’t forget to subscribe to the newsletter below so you don’t miss the next post (and get some free gifts I prepared for you) πŸ™‚

ruby ranges

Ruby Ranges: How Do They Work?

Have you ever wondered how ranges work in Ruby?

Even if you haven’t, isn’t it fun to discover how things work under the hood?

That’s exactly what I’m going to show you in this post!

Understanding Ranges

Just as a reminder, this is what a Ruby range looks like:

The parenthesis are not necessary to define a Range, but if you want to call methods on your range you will need them (otherwise you are calling the method on the 2nd element of the range, instead of the range itself).

The Range class includes Enumerable, so you get all the powerful iteration methods without having to convert the range into an array.

Range has some useful methods, like the step method.

Example:

Other Range methods to be aware of are: cover? & include?. It would be a mistake to think that they do the same thing, because they don’t.

The include? method just does what you would expect, check for inclusion inside the range. So it would be equivalent to expanding the Range into an Array and checking if something is in there.

But cover? is different, all it does is check against the initial & ending values of the range (begin <= obj <= end), which can yield unexpected results.

Example:

The cover? example is equivalent to:

The reason this returns true is that strings are compared character by character. Since “a” comes before “c”, the characters that come after the first “c” don’t matter.

Range Implementation

Ranges are not limited to numbers & letters, you can use any objects as long as they implement the following methods: <=> and succ.

For example, here is a time range:

So how does this work? Let’s take a look at this implementation:

I added some comments to help you understand what is going on. The idea is that we keep calling the next method on the first object until it is equal to the second one, the assumption is that they will eventually meet.

Custom Class Ranges

Most of the time you will be using number & character ranges, but it’s still good to know how you can use ranges in a custom class.

Example:

The key here is to make sure that you implement the <=> & succ methods correctly.

If you want to use the include? method you need to include the Comparable module, which adds methods like ==, <, and > (all based on the results of the <=> method).

Conclusion

In this article you have learned how ranges work in Ruby so you can understand them better & implement your own objects that support range operations.

Don’t forget to subscribe to the newsletter below so you don’t miss the next post πŸ™‚

ruby standard library

5 Useful Examples From The Ruby Standard Library

The Ruby Standard Library is a series of modules & classes that come with Ruby but are not part of the language itself.

These classes offer a variety of utilities like Base64 encoding, prime number generation & DNS resolution.

In this article I’m going to show you 5 of these classes with useful examples.

Unique Items with Set

Set is a data structure that guarantees unique items.

Here is an example:

The implementation of this data structure is based on the Hash class, so there is no Array-like indexing.

But if you just need a bucket of data where all elements need to be unique all the time, then the Set might be what you are looking for.

Logging Messages

If you need to log some error / debug message Ruby has you covered with the Logger class. This class provides everything you need to start logging.

In fact, this class is what Rails uses by default.

To use the Logger class you can simply create a Logger object & give it an output stream (or a file name) as a parameter. Then you can use the different logging levels to register your message.

The logging levels are: debug, info, warn, error & fatal.

Example:

This produces the following output:

The first character is an abbreviated form of the logging level (I for Info, W for Warn)… Then you have a timestamp, and the current process id (which you can get in Ruby using Process.pid).

Finally, you have the full logging level & the actual message. You can change this format by providing a new formatter.

Working with Prime Numbers

Maybe you don’t need to deal with prime numbers on a day-to-day basis, but it’s still good to know (specially if you like programming challenges) that Ruby has good support for them via the Prime class.

When you require Prime it will add the prime? method to Fixnum.

This class also includes a prime number generator.

Example:

Using StringIO

The StringIO class allows you to create a string that behaves like an IO object. This means that you can work with this string like if you were reading from a file or STDIN (Standard Input).

Here is an example:

Notice a few things: when you add data into your StringIO object it will not add spaces or newlines for you, and to get the actual string you need to call the string method. Also you can’t use array indexing to access individual characters, like you can with a regular string.

When is this useful? Well, sometimes you may want to substitute a file or some other IO object for another object that you have more control over. For example, in a testing environment you can replace STDOUT with a StringIO object.

You can see a real world example from Rails here: https://github.com/rails/rails/blob/52ce6ece8c8f74064bb64e0a0b1ddd83092718e1/activesupport/test/logger_test.rb

Working with Paths

The Pathname class wraps several file-system exploring utilities, like Dir & File, in a much more powerful class.

While the method names are the same, they return Pathname objects instead of strings or arrays. And what this means is that you can keep working with the results with all the file-related methods.

This is a great example:

If you tried to do this with Dir, you would have to use this code, which is not as elegant.

Give this one a try the next time you need to do a file-system related task πŸ™‚

Conclusion

I hope you found these examples useful! Make sure to explore the Standard Library a bit more so you can learn what it can do for you.

Don’t forget to share this post if you liked it πŸ™‚

weighted random numbers in ruby

How to Generate Weighted Random Numbers

Random numbers usually follow what we call a ‘uniform distribution’, meaning that there is the same chance that any of the numbers is picked.

But if you want some numbers to be picked more often than others you will need a different strategy: a weighted random number generator.

Some practical applications include:

  • the loot table in a video game, where enemies can drop different items with varying drop rates.
  • a raffle, where people with more tickets have more chances to win.

Simple Strategy

If you think about the raffle example you can come up with an obvious solution: generate an array which has one copy of the item for each ‘ticket’.

For example, if John buys 4 raffle tickets and David only buys 1, John will have 4 times more chances to win than David.

Here is a working implementation:

I’m adding the person’s name once for every ticket they bought, and then I pick a random name from that list. By virtue of being on the list more times, it will increase the chances of that name being picked.

I like this approach because it’s very simple and once you have your list it’s very fast to pick a winner.

Sum of Weights

There is another way you can do this that is more memory-efficient, the trade-off is that picking a random value is slower.

The idea is to pick a random number between 1 and the sum of all the weights, and then loop until you find a weight that is lower or equal than this number.

Here is the code:

This code takes in a hash where the keys are the items and the values are the weights. You can call this method like this:

You can test if this works as expected by looking at the distribution of the results after running it many times.

Here is an example:

Run this a few times and look at the output to see if the ratio is what it should be.

Conclusion

While there are more sophisticated algorithms, these two should serve you well. I hope you found this article useful, please share it with your friends so I can keep writing more!

metaprogramming ruby

Metaprogramming in The Wild

You may have read about Ruby metaprogramming before & maybe you have used it in some of your projects, but how are some of the most popular open-source projects making use of this feature?

Find out in this post!

Rails Example

Rails makes heavy use of metaprogramming, so it’s a good place to start looking.

For example, when you want to check the current environment in your Rails app, you do something like:

But what is env? And how does that work? The answers are in the StringInquirer class, which is a subclass of String. This class uses method_missing so that you can call env.production? instead of env == production.

This is what the code looks like:

This is saying: “if the method name ends with a question mark then do the comparison, otherwise keep going up the ancestors chain”.

The original code can be found here.

Sinatra Delegation

If you have used Sinatra before you may know that there are two ways to define your routes: by using the get / post methods directly, outside of any class. Or by defining a class that inherits from Sinatra::Application.

The Sinatra DSL (Domain-Specific Language) methods are defined inside Sinatra::Application, so how can you use them outside of this class?

Well, there are two things going on here: metaprogramming & module extension.

Sinatra defines a Sinatra::Delegator module inside base.rb, which is used to delegate method calls to a target. The target is set to Sinatra::Application by default.

This is a simplified version of the delegate class method, defined in Sinatra::Delegator:

This code is creating a set of methods (with define_method) that will forward the method calls to Sinatra::Application (the default value for Delegator.target).

Then the main object is extended with the methods defined in Sinatra::Delegator, which makes the Sinatra DSL available outside the Sinatra::Application class.

It’s worth noting that Ruby has two built-in classes for method delegation, in case you need them: Delegator & Forwardable.

The Paperclip Gem

Paperclip is a gem which allows your application to handle file uploads. Files are associated with ActiveRecord models. To define an attached file on a model you can use the has_attached_file method.

For example:

This method is defined on paperclip.rb and it uses metaprogramming to define a method on the Model. This method can be used to access the file attachment (which is an instance of the Paperclip::Attachment class). For example, if the attached file is :avatar, Paperclip will define an avatar method on the model.

Here is the define_instance_getter method, which is responsible for that:

From this code we can see that the attachment object is stored under the @attachment_#{name} instance variable. On our example that would be @attachment_avatar.

Then it checks if this attachment already exists, and if it doesn’t, it creates a new Attachment object and sets the instance variable.

Here is the original source code.

Conclusion

As you have seen, metaprogramming is a very powerful and flexible technique, but whenever you want to reach for it remember that quote that says: “With great power comes great responsibility”.

If you enjoyed this post don’t forget to subscribe to my newsletter πŸ™‚