UNIX Utility Recreation, cat: Part One

[link—standalone]

The first utility I decided to recreate was cat. While I am sure that the overwhelming majority of people reading this post are familiar with how cat works, however in the interest of padding the word count I'll explain it anyway. Cat is a shell utility that concatenates the contents of one or more files, and displays that onto stdout, it also works with input from standard in. It is an extremely simple utility, almost as simple as echo. This simplicity is the reason I chose to replicate cat first, as it seemed to be the most approachable of the shell utilities from a design standpoint. It also has quite a few options and flags that while simple, may prove to be an interesting challenge. I will go over those options in the next part, for now let's just focus on the most basic functionality.

In this case, the basic functionality can be broken down like this:

1. Check the argument list
1. If no arguments are encountered, echo stdin
2. If arguments are encountered, move to step 2
2. Parse the arguments by iterating over them
1. if a flag is encountered, process it (in this case "processing" just means skip)
2. if a file name or "-" is encountered, add it to the array of files to concatenate
3. iterate over the list of files to concatenate, add echo stdin when "-" is encountered
4. output the contents of each file concatenated
5. exit

I made a few simplifications to this process in my implementation, as we will see shortly. My implementation gives the same results as GNU cat with no flags, verified by sending the output to a file from each and running diff on the files, but has some issues handling signals, such as CTRL-D, which in GNU cat signals to quit echoing stdin and move to the next file in the list. I hope to resolve this issue before the next stage of this utility, and may have to examine the source of gnu cat earlier than anticipated to ensure I am doing things right. This is a learning exercise after all, so I am not too upset about it if I do. I just don't want to make that a habit, so I'm not just copy pasting what already exists.

Let's get into the meat and potatoes of my attempt at recreating this iconic utility. The first feature that I implemented was simply echoing input from stdin to stdout. This function was trivial, all I did was use fgets into an input buffer. I have a defined buffer size that the input buffer is set as, and that fgets uses to check input. This will fail in an input is received that is larger than the buffer size.

Next, I implemented simple file output. My solution is extremely simple, and is included here so that we may all mock it together.

int simpleCat(const char * filename) {
	FILE *input_fp;
	char line_buffer[BUFFER_SIZE];
	char *stat;
	const int chars_to_read = BUFFER_SIZE;
	input_fp = fopen(filename, "r");
	if (input_fp == NULL) {
		printf("Unable to open file: %s", filename);
		return -127;
	}

	while(1) {
		stat = fgets(line_buffer, chars_to_read, input_fp);
		if (stat == NULL) {
			if ( feof(input_fp) != 0) {
				break;
			} else {
				printf("Unable to read from file: %s", filename);
				return -126;
			}
		}
		fputs(line_buffer, stdout);
	}

	return 0;
}

As you can see, this is an extremely brutal solution, all it does is simply open a file, ensure the file was opened correctly, then parse the file line by line and output each line once a good read is confirmed. Then terminates when EOF is reached. I also have some return codes, however this is so simple that there is no real need for them, as it will just exit after printing the error.

Finally we can take a brief look at how I am handling arguments. Again, it is an extremely naive and simplistic solution.

int main(int argc, char* argv[]) {
	if (argc > 1 && strcmp(argv[1], "-") != 0) {
		size_t optind;
		int i = 0;
		for(optind = 1; optind < argc; optind++) {
			if (argv[optind][0] == '-' && strlen(argv[optind]) == 1) {
				echoStandardIn();
			} else if (argv[optind][0] != '-')  {
				int stat = simpleCat((const char*) argv[optind]);
				if (stat != 0) {
					exit(stat);
				}
			}
		}
	}
	else {
		echoStandardIn();
	}
	return 0;
}

This most certainly won't win any awards for elegance, however it does get the job done. Now you can see the simplification I made to the functionality stated above. Instead of handling the flags, and shunting the file list into a temporary array, we simple treat argv as the file list, and skip over anything that looks like a flag. This will no work in the second stage, as I will have to know what flags have been sent, and how that affects my output once I reach that step. However, this solution performs as expected in the tests that I have run. I am sure there is something I am missing, however, I am pleased with my results thus far.

Conclusion

This project is going smoothly, but after just this little bit, I have seen some issues as far as validation and testing. Ideally, I would like to have a test suite that looks for things like buffer overflows and other weird issue, as well as automates validating my outputs with those of the GNU utilities. But I have no idea how to write something like that currently. I am going to look at testing code like this and try to recreate them before moving on to the next stage. This early I am comfortable taking breaks to make my life better in the long run. Also the signal issue has made me realize that I might be missing some key knowledge for developing shell utilities like this, so I will also be on a knowledge hunt. If anyone has suggestions for a decent testing framework, or were I can find knowledge about how shell utilities are supposed to talk to the OS as a whole, feel free to contact me and let me know.

Anyway, thank you all for reading and I hope to see you in the next part!

Sat, 29 Aug 2020 13:11:45 -0500

UNIX Util Recreation Intro

[link—standalone]

This is a simple project in which I attempt to recreate the GNU core utilities. In order to keep myself fresh while working from home, I have decided to take a deep dive into the UNIX shell utilities (technically the GNU core utils). I am recreating them in no particular order (read: whatever seems easiest first) to help myself better understand C and this sort of development. The way I am planning on going about this is pretty simple, I pick a utility, run 'man [utility]' and see if I can replicate the functionality myself. There are a few caveats, I am allowing myself to use system libraries, unless the functionality is interesting or novel (read: I passed university, I can write strcmp and I don't feel like doing that every project, an example of interesting functionality would be argument parsing). Also, I am going to do my best to keep things in single modules instead of splitting them up into libraries, this is just so I can get really get a feel for ** all ** the moving parts of each utility (this is why I will be using system libraries for things like strcmp). Finally, everything will be compiled with clang, not GCC. I don't have any particular reason for this other than I like clang.i

Every utility I do will have three stages. The first stage I will implement the simplest version I can of the utility. This means base functionality only, no flags, just whatever the core feature is (example, for the first stage of 'cat' I only do stdin echoing and file output). The second stage I will attempt to implement all of the options and flags of the command. The final stage I will compare my produced code with the source code from the GNU source utils, and write up the differences. Each stage will have its own write up, so that I can reflect on what I did, why, and perhaps correct my mistakes. I hope that this project will be beneficial to me in the long run, allowing me to grow as a programmer, as well as giving me some practical experience before jumping into contributing to open source projects like suckless software, or maybe even eventually contributing to linux kernel code.

Fri, 28 Aug 2020 15:53:13 -0500

Cynical Software

While I was still in school I had the opportunity(?) to participate in a hackathon. For those unfamiliar (and to pad my word count), a hackathon is a coding competition where you or a small team produce some software that is then judged by some people who judge software. Some of these have themes, and some do not, and the one that I was in did not. My team decided that we would address what we perceived as an issue in the IT / System Administration world, namely bloated ticketing tools. We brainstormed the problem and eventually came to the conclusion that the issue with most ticketing software is not the main feature, i.e. ticketing, but the clunky GUI that comes along for the ride. We deduced that the GUI is not made for the sysadmin or programmer or what have you, but the managers and admin staff, people who want to make sure the code monkeys are working. However, the GUI tends to lead to an anti-pattern, where a programmer or sysadmin spends most of there time hopping around in shells, or IDE's, but then having to spend more time logging their actions in the GUI. We decided that to fix this, we would right a lightweight shell utility, that could exist on any machine, and log tickets to the ticket service database, and the clicky-clacky GUI could see them. Thus Terminal Ticket was born. It targeted JIRA, since it is very widely used. We built the thing, and came up with a demo and... Came in dead last.

Now, there are a few reasons for this (other entries were way better, we were all first timers, etc), but the one that I want to focus on today is the fact that it was not cynical enough.

By cynical, I mean the school philosophy, the one the OG mad lad Diogenes was a student of. Cynicism al la Diogenes is the rejection of all things unnecessary in life. Diogenes was an extreme example, living in a barrel in the market, urinating in the street, and allegedly throwing away his only bowl after seeing a small boy drinking from his hands. Our software was not cynical and did a lot of unnecessary things. For one thing, we wrote it python, allegedly to "speed development" which, in a way it did because it allowed us to develop a lot of junk in a short amount of time. Python carries a large amount of extras that must be bundled onto a system, so there goes our lightweight requirement. Python is also pretty slow, not GUI slow but still slow in terms of shell, and this was on top of api access, and a complicated configuration to plug into you enterprise's JIRA service and a hundred other things. It also did more than just opening and closing tickets, a lot more. All of this bloat made our presentation confusing at best, and only one judge really "got it."

So, we lost, but I refused to believe it. I railed against the Academic judges. I thought we had produced the next VIM, and that sysadmins everywhere would bow before our superior product. So I made a kick starter, and made 1 dollar. Then I finally gave up, and left it alone for awhile. But it stayed in the back of my mind. It was not until I read this article, from Suckless Software's website, that my eyes were opened. I felt like Diogenes must have upon seeing the child drinking from his hands. "What a fool I am!" I exclaimed, as I beheld the majesty of the erlang ticket system. One command, everything tracked through git, it was beautiful, it was pure genius. I realized the flaw in Terminal Ticket was that we did too much that was unnecessary. We were not geniuses, we were falling to the enterprise trap of doing everything, and the judges could see it. What we had made was a sucky piece of junk that was only "minimal" because it was on the command line.

The cynics sought to remove unnecessary aspects of life, and to live without them. We thought we were removing unnecessary bloat, because we were working around the GUI. In reality, all we did was drag all the bloat into a command line interface, and it was confusing and slow. In reality, we should have been more cynical and focused on only the feature we needed, instead of trying to integrate every feature of the service we were using.

The point of this embarrassing tale of woe is that if you want to make truly suckless software, you have to be a cynic. Remove what is truly unnecessary, especially if it is handled by existing tools. Minimize, minimize, minimize. If you can not do that, then just opt to make something that sucks but looks flashy, because something that sucks but is pretending to minimal will be dropped in a heart beat by the people and programmers who know what they are doing. To round this story out, I am going to share that script I use to track time on personal projects. I developed it in the spirit of the Erlang Ticket system, although mine is only for a single dev, namely myself.

#!/bin/bash

DIR=$(pwd)
TOOLD="/home/kevin/Projects/Python/ticket"
TFILE=$DIR/tickets
TEMPLATE=$TOOLD/template
TEMP="$TOOLD/.tmp"

open() {
	branch=$(git branch --show-current)
	git checkout ticket_tracking
	read -erp "Ticket Title: " title
	ticketnum=$(grep -o \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* tickets | wc -w)
	ticketnum=$(expr $ticketnum / 2)
	ticketnum=$(expr $ticketnum + 1)
	cat $TEMPLATE > $TEMP/tmp
	sed -i "s/&l tide >/$ticketnum/g" "$TEMP/tmp" 
	sed -i "s/< title >/$title/g" "$TEMP/tmp" 
	$EDITOR $TEMP/tmp
	cat $TEMP/tmp > > $TFILE 
	rm $TEMP/*

	git add $TFILE
	git commit -m "Added new ticket with id $ticketnum and title $title"
	git push origin ticket_tracking
	git checkout  $branch
}

open

				

That is it. I run this when I start a new task, and when I finish, I open the ticket file, mark the corresponding ticket number as i closed and commit it to the tracking branch. Then I can easily used the commit history on that branch to track time on each task. It it simple elegant, and fit's in very well with my current work flow. Would it work for anyone else? Probably not. But the point is not for it to be useful to everyone, it's to make my life better, and reduce unnecessary work.