Hacker Public Radio

This show has been flagged as Clean by the host.

In his early days as a writer, Heinlein wrote his stories in the context of a shared universe that he called the Future History. These were mostly short stories at first, with the occasional novella. But they include some great stories.

The Future History, Part 2


There were a few key themes running through Heinlein’s body of work. One we have already remarked upon, individual freedom, which had to be protected from any source of power, including both government and private corporations. This was essentially a libertarian perspective, but unlike many of today’s libertarians he was equally averse to the corporate type of power as a threat. But he had a complex view of the world which has resisted some attempts to pigeonhole him. He started out as a socialist, and while he didn’t remain one, he never became a knee-jerk reactionary either. In fact, he clearly despised them just as much. One way of looking at his body of work is that he explored the ramifications of different social policies through his stories, but in most cases the needs of a good story came first in the early years. In his later works he often surrendered to the temptation to pontificate, which reduced the enjoyment of them somewhat for anyone who was not already in agreement with his opinions

The second major theme you see throughout all of his works is the idea of the competent individual. He admired anyone who could do a job well, and clearly did not care whether they were man or woman, nor black or white. Alexei Panshin writes, in Heinlein in Dimension:

“There is one unique and vivid human Heinlein character, but he is a composite of Joe-Jim Gregory, Harriman, Waldo, Lazarus Long, Mr. Kiku, and many others, rather than any one individual. I call the composite the Heinlein Individual. . . . It is a single personality that appears in three different stages and is repeated in every Heinlein book in one form or another.

“The earliest stage is that of the competent but naïve youngster. . . . The second stage is the competent man in full glory, the man who knows how things work. . . . The last stage is the wise old man who not only knows how things work, but why they work, too.”

Harriman we have already encountered in The Man Who Sold The Moon, and the others appear later. The Heinlein Individual, as he is often referred to, appears in many of Heinlein’s stories.

A third major theme has to do with morality and religion. Heinlein grew up in what he considered the heart of the Bible Belt, in Missouri, and saw first-hand how the evangelical Christians operated, and despised what he saw. As someone who believed in individual freedom, he could never surrender to someone else’s idea of how he should live his life. He saw them as a danger to his ideal libertarian society, and this shows up very early in his work. He personified the good, upright, church-going folk as “Mrs. Grundy”, and while you might want to draw the drapes to keep her from knowing what you were doing, you should never let her dictate how you would live your life. Revolt in 2100 begins the exploration of this in detail.

There is a chart of the future history at Baen Books, and in it we see that the 1960s were what Heinlein called The Crazy Years. (Remember, he conceived this in the 1940s and 1950s.) But in 2012 the major thing occurred when Nehemiah Scudder, a backwoods preacher, managed to get elected as President. This would be the last election held under the U.S. Constitution as he established a religious dictatorship that lasted a couple of generations. IS this plausible? Heinlein wrote about this:

“As for … the idea that we could lose our freedom by succumbing to a wave of religious hysteria, I am sorry to say that I consider it possible. I hope that it is not probable. But there is a latent deep strain of religious fanaticism in this, our culture; it is rooted in our history and it has broken out many times in the past.

“It is with us now; there has been a sharp rise in strongly evangelical sects in this country in recent years, some of which hold beliefs theocratic in the extreme, anti-intellectual, anti-scientific, and anti-libertarian.“

His background in the Bible Belt is what informs a lot of his thinking. He goes on to describe how this might happen:

“Throw in a Depression for good measure, promise a material heaven here on earth, add a dash of anti-Semitism, anti-Catholicism, anti-Negroism, and a good large dose of anti-“furriners” in general and anti-intellectuals here at home, and the result might be something quite frightening — particularly when one recalls that our voting system is such that a minority distributed as pluralities in enough states can constitute a working majority in Washington.”

As the science fiction author David Brin points out, Heinlein accurately predicted much of what we are going through in the United States right now. There is an emerging dictatorship in the United States, promoted by right-wing religious groups. The “material heaven here on earth” is represented by the Prosperity Gospel, prominent in the Trump movement, and so on. Where the Prophet used a restored Ku Klux Klan as his muscle, we have The Proud Boys, and so on. It really does track very closely. Read David Brin’s article for more on this.

But nothing lasts forever. Empires rise and fall, governments change, and in this case a resistance movement arises. The revolt is depicted in the novella If This Goes On— (1940), and it is set in the year 2100, giving the title to the book. The main character is John Lyle, who is a young army officer assigned to the group protecting The Prophet in his capital of New Jerusalem. In the beginning he is thoroughly indoctrinated, but then begins to question his beliefs when he falls for one of The Prophet’s virgins, Sister Judith. He has an older companion in the military who is not only unshocked when John confides in him about his doubts, but offers to help him. It turns out this companion, Zeb Jones, is a member of the underground group called The Cabal that is working to overthrow the theocracy. In the end they are successful, and in the course of this John Lyle does a lot of growing up. In this we see another common characteristic of Heinlein stories: a young, naive boy meets up with an older and wiser man who helps him to grow.

In 2016 If This Goes On— won the Retro-Hugo Award for best novella of 1940. And in a personal note, I have T-shirt that says “Scudder for President 2012”. This baffles most people, but I enjoy the in-joke.

What is interesting in this book is that Heinlein doesn’t stop with a successful revolution. He then goes on in a second novella to describe the government that arose following the revolution, and this story is called Coventry (1940). The new government that arises after the revolution is called The Covenant, and it is an attempt to make sure that what happened with Scudder in 2012 could never happen again. It is a strongly libertarian government based on an agreement to be non-violent. In this society, scientists can cured criminal or violent tendencies, but any citizen convicted of such must agree to the treatment. The alternative to treatment is that they can be exiled to a place called Coventry. Coventry is outside of the Covenant society, and the Covenant society has nothing to do with them.

Our protagonist, David McKinnon, is convicted of assault, and chooses to go to Coventry instead of getting treatment. He imagines it is a peaceful anarchy, but is disabused of this notion when he is robbed of all of this possessions upon entry and thrown in jail. A fellow inmate, Fader Magee, helps him escape, and we learn he is an agent of the Covenant government. They learn that two of the factions in Coventry have joined forces, and found a way to break through the barrier that surrounds Coventry. They plan to attack and overthrow the Covenant government. David and Fader separately work to escape and get back to warn the Covenant government, which they do successfully. And by doing this, David has demonstrated that he is no longer a danger to the Covenant society and no longer subject to treatment.

This story won a Prometheus Hall of Fame Award, which is awarded by the Libertarian Futurist Society. And the Covenant society certainly has libertarian features. But this is not the Randian version of libertarianism, as exemplified by the fact that David is restored to the society because he demonstrated his concern for others. Heinlein always promoted individual freedom, but also the idea that people have a responsibility towards others.

Finally, Revolt in 2100 contains the short story Misfit, w2hich we have looked at previously.

The Past Through Tomorrow (1967) is a one volume collection of most of the Future history stories. I say most because just which stories belonged in this group could change from time to time. It also has the last version of the Chart of the Future History, and a few stories we have not yet mentioned (Methusaleh’s Children, and The Menace From Earth). And many of his other works contain back references to these events that imply that they might be set in the same alternate universe. Heinlein gets the last word on this:

“I have never been sure whether or not publishing that chart was a good idea or a bad mistake. Possibly it helped to sell some stories later—but certainly it caused me and still causes me to receive a lot nuisance mail from nitpickers. I have never felt bound by that chart; it was to serve me, not the other way around. If I found myself with a good story notion which fitted fairly well into the chart but not perfectly, I shed no tears—I went ahead and let the inconsistencies stand.

I want each story to be internally consistent . . . but I won’t let myself be painted into a corner through trying to fit that chart perfectly. I may start another “Future History” story tomorrow . . . and find that to make it a good yarn I must violate some item on that chart. I’ll give the nitpickers something to pick, for I will not hurt a good yarn for the sake of “logic”—logic is not involved, as that chart is fiction, not Holy Writ.”

Links:

https://www.amazon.com/Heinlein-Dimension-Critical-Alexei-Panshin/dp/0911682015

https://web.archive.org/web/20151105170345/http://www.baenebooks.com/chapters/1439133417/1439133417___1.htm

https://david-brin.medium.com/heinleins-future-history-coming-true-before-our-eyes-10356a95556a

https://en.wikipedia.org/wiki/%22If_This_Goes_On%E2%80%94%22

https://en.wikipedia.org/wiki/Coventry_(short_story)

https://en.wikipedia.org/wiki/Prometheus_Award#Hall_of_Fame_Award_inductees

https://en.wikipedia.org/wiki/The_Past_Through_Tomorrow

https://www.palain.com/science-fiction/the-golden-age/robert-a-heinlein/the-future-history-part-2/

Provide feedback on this episode.

This show has been flagged as Clean by the host.

Kevie, Paul and Dave are back once again with the HPR Beer Garden, this time focusing on the relatively new style of beer called Double IPA. Kevie samples
Man Overboard
by
Cromarty Brewing
, Paul tastes
Rumble
by
Beavertown
and Dave opts for
Ultra Phase by North Brewing














Connect with the guys on Untappd:







Dave






Paul






Kevie










The intro sounds for the show are used from:







https://freesound.org/people/mixtus/sounds/329806/






https://freesound.org/people/j1987/sounds/123003/






https://freesound.org/people/greatsoundstube/sounds/628437/










The next 3 beer styles to be reviewed:





Belgian Blonde



DDH IPA



Amber Ale



Provide feedback on this episode.

This show has been flagged as Clean by the host.





01 Introduction







In this episode I will describe how to calculate elapsed time in bash or other shell scripts.



While this may sound like a very simple and basic thing to do, there is a slightly more complex aspect to it if you wish to calculate elapsed time to a higher resolution than one second.







02



There are many reasons for calculating elapsed time in a shell script.



For example you may wish to simply report how long an operation took to run.



Another reason may be that you are trying to speed up a script and need to calculate benchmark data to see how different alternative methods perform.







03



What may seem like a simple task gets a bit more complicated if you want to do it for multiple different operating systems even if they are all unix related, as we shall see.







--------------------







04 Operating Systems Tested







For the purposes of this episode, I ran tests on the current version of the following operating systems.







Alma



Alpine



Debian



FreeBSD



OpenBSD



RaspberryPi



OpenSuse



Ubuntu 2604







Alma is a close copy of Red Hat that we can take as representing Red Hat style distros.







--------------------







05 Simple Low Resolution Timing







I will start with the simple and obvious method before describing the less obvious ones.







This uses the date command to get the current time in seconds since the unix epoch.



This is simply



date '+%s'







06



Save this to a variable using whatever method you prefer.



For example.



starttime=$(date '+%s')







07



Next, do whatever operations it is you wish to time.



Use the date command to get the current time again.



endtime=$(date '+%s')







08



Now simply subtract the start time from the end time using shell arithmetic.



This should be very obvious and basic.







--------------------







09 Higher Resolution Timing







However, suppose we wish to measure time to greater than one second of precision.



We need to do two things.



The first is to obtain the current time at a higher degree of precision.



The second is to conduct the calculations to a higher degree of precision.







10



Unfortunately, the standard time precision for POSIX shells seems to be 1 second.



Some shells offer a higher precision, but others do not.



Furthermore, standard shell arithmatic uses integer, which limits calculations to 1 second of precision.







--------------------







11 Bash High Resolution Shell Variable







Fortunately, bash is one that does offer a high precision date.







If you are using bash 5.0 or newer, there is a shell variable called EPOCHREALTIME which offers time since the the unix epoch (that is, since the first of January 1970, at 00:00:00 UTC) in seconds to 6 decimals of precision.







12



Example



echo $EPOCHREALTIME



1779634800.184926







13



This is related to the similar bash variable known as EPOCHSECONDS which gives the number of seconds since the unix epoch.







14



Example



echo $EPOCHSECONDS



1779634800







15



So if you are using bash, measuring time is very simple.







--------------------







16 But is it Really Bash?







Is your script however actually using bash?



Debian and derivatives actually have two shells.



The first, the interactive shell is bash.



The second, the non-interactive shell is dash, which stands for "debian almquist shell".







17



If you open a terminal, you get bash.



If your script starts with a "bin/bash" shebang line, you get bash.



However, if your script starts with a "bin/sh" shebang line, you get dash.



Some people find themselves getting caught out by this one when they try something out in a terminal but find that it doesn't work in their script which started with "bin/sh".







18



Many other, but not all, Linux distros use bash for both the interactive and non-interactive shells, so "bin/sh" and "bin/bash" work the same with those ones.







So if you intend to use bash, make sure your script calls for bash in the first line.







--------------------







19 The SHELL Variable







So how can a script tell what shell it is running under?



There is a shell variable called "SHELL" which will tell you the name of the shell.



Well, sort of.







20



On Debian and derivatives "SHELL" will say "bash" regardless of whether the actual shell is bash or dash.



On some other operating systems "SHELL" will simply say "sh" even if it is something else entirely.







So we need to do some additional levels of checking to see what we have.







21



To start with though, here's what each of the test distros reports for SHELL.







Alma : bash



Alpine : sh



Debian : bash



FreeBSD : sh



OpenBSD : ksh



Raspberry-Pi : bash



Suse : bash



ubuntu2604 : bash







--------------------







22 Bash Versus Dash







First, let's try to see which ones are bash and which ones are dash.







The first thing we can check is for the shell variable BASH_VERSION.







23



Example



echo $BASH_VERSION







If the shell is bash, then it will report a version string.



If the shell is not bash, then it will return an empty value.







24



Using this test, we can see that Alma and Opensuse are indeed using bash.







We however need to check Debian, Raspberry Pi, and Ubuntu when running in an "sh" script.



To check this we can use the "which" command to see what "sh" actually is.







25



Example



echo $( ls -l $(which sh ) | rev | cut -d" " -f1 | cut -d/ -f1 | rev )







26



"which sh" shows us the path to "sh"



However, that is a link so we need to use



"ls -l" to find the actual executable.



"rev" reverses the string.







27



"cut" takes the first element separated by spaces.



The second



"cut" takes the first element separated by the "/" characters.



The final "rev" takes that string and reverses it again to get it in the correct order.







28



In the case of Debian, Raspberry Pi, and Ubuntu it tells us that this is "dash".







--------------------







29 Openbsd







Openbsd reports its shell as "ksh", which stands for Korn Shell.



It is indeed Korn Shell, so we simply leave that one as is.







--------------------







30 Alpine and Freebsd







Next we have Alpine Linux and Freebsd, which both report as "sh".







In the case Freebsd there doesn't appear to be any further we can go that I am aware of.



It's simple "sh".



It is a basic POSIX shell which seems to be similar to the original unix shell, the Bourne Shell.



Older versions of Freebsd used a different shell known as tsch (the C shell), but I haven't tested that so I will ignore that here.







31



With Alpine Linux however, we can get the actual shell using the same method that we used for Debian Linux.



This reports as being "busybox".







32



Busybox is a limited shell intended for use in embedded systems.



Alpine was originally an embedded distro, but some people started using it for containers.



Alpine is Linux, but it is not GNU/Linux, and there are a number of areas which can trip you up if you are not aware of them.



So, be extra careful if you are using it for anything, and test everything.







--------------------







33 Summary of Actual Shells







Here is our revised list with the actual shell used when asking for "sh", so far as we can determine.







Alma : bash



Alpine : busybox



Debian : dash



FreeBSD : sh



OpenBSD : ksh



Raspberry-Pi : dash



Suse : bash



ubuntu2604 : dash







34



There are other shells, but none of them are the default shell for any of the distros on our list, so I haven't tested them.







--------------------







35 Solutions for Measuring Time







Now we need to find solutions for bash, dash, ksh, sh, and busybox.







--------------------







36 Bash







For bash, we can simply use EPOCHREALTIME, as mentioned above.







--------------------







37 Dash







For dash, we can use the date command.



This is a very conventional method, and is probably the first answer that anyone would give for this situation.



However, while it will work in most cases, it will not work in all cases, so it is not a universal solution.







38



To use date we simply call it with the correct format string.



This uses %s to get seconds since the epoch, and %N to get nanoseconds of the current second.



If you put a decimal separator between the two it will appear in the output.



You can use the correct decimal separator for your locale, but I won't go into that here.



Instead I will just assume a period or dot.







39



Example



date '+%s.%N'



1779634800.358916385







--------------------







40 Problems with Date on Alpine and Openbsd







Date will work for bash, dash, and sh on Freebsd.



However it will not work for ksh on Openbsd, or for busybox on Alpine.







41



With busybox on Alpine, it simply ignores the %N format specifier and prints out the epoch in seconds only followed by the decimal separator.



=



Example



date '+%s.%N'



1779634800.







42



With ksh on Openbsd it prints the epoch in seconds followed by the decimal separator and then the %N as a literal N.







date '+%s.%N'



1779634800.N







43



Fortunately we have alternatives for these two cases.







--------------------







44 Openbsd







Openbsd has the "ts" or timestamp utility installed by default.



ts prints a time stamp in front of every line it receives from standard input.



I won't go into details on all aspects of ts here, I'll leave that to someone else.



Instead I will focus on how to use it for our specific purposes here.







45



We need to provide a format specifier to ts, which in this case is "%.s"



We also need to provide something for standard input, or otherwise ts will simply sit there and wait for input.



So what we need to do is to echo nothing through a pipe to ts while also giving ts the proper format specifier.







46



Example



echo | ts "%.s"







47



This will output the epoch time in seconds to six decimals of precision.







ts is installed in Openbsd and Freebsd by default and can be used in either.



It can also be installed in many other distros.











--------------------







48 Busybox on Alpine







None of the methods discussed so far will work for busybox on Alpine though.



However there is a way, but it's a bit non obvious and somewhat hacky.







49



Busybox includes a command called "adjtimex".



This is normally used to adjust the time hardware.



However if it is run without arguments, it will report the current settings.







50



These include the current epoch time in seconds , and in another field the time in microseconds.



These are reported as key value pairs.



So what we need to do is to do the following







51



Run adjtimex



Capture the output.



Grep for "time.tv_sec"



Grep for "time.tv_usec"



Use cut to extract the time value in each case.



Use tr to get rid of excess spaces in each case.



Combine the two in a string with a decimal separator between them.







52



This takes a total of 4 lines of shell script.



I will just describe them breifly here, see the show notes for details.







53



First we want to capture the output of adjtimex in a single operation.



Run adjtimex and pipe the output through grep to capture lines containing "time.tv_"



and save this to a variable.







# Extract the current high resolution time from adjtimex.



# We want two key value pairs, identified by time.tv_sec and time.tv_usec.



tvals=$( adjtimex | grep "time.tv_" )







54



Next echo the contents of this variable and pipe it through grep, cut, and tr to get first the seconds and then the microseconds while also removing excess spaces.



Save these to two separate variables.



"time.tv_sec" is the time in seconds since the epoch.



"time.tv_usec" is the number of microseconds in the current second.







# Get the time since the unix epoch in seconds and micro-seconds.



timesec=$( echo "$tvals" | grep "time.tv_sec" | cut -d: -f2 | tr -d " " )



timeusec=$( echo "$tvals" | grep "time.tv_usec" | cut -d: -f2 | tr -d " " )







55



Adjtimex does not zero pad the microsecond time value to provide leading zeros, so we need to take care of this using printf before we can append it to the seconds value. We didn't need to do this with date where the %N format character does this automatically.







In this instance, the printf format string is '%06d'







padusec=$( printf '%06d' $timeusec )







Now, combine these into a single number with a decimal separator by using simple string concatenation.



# Combine them into a single number.



timehires="$timesec"".""$padusec"











--------------------







56 Summary of Methods







Let's summarize where we are so far in terms of methods we can use to get the current time as a high resolution number.







Alma : use EPOCHREALTIME or date



Debian (bash) : use EPOCHREALTIME or date



Raspberry-Pi (bash) : use EPOCHREALTIME or date



ubuntu2604 (bash) : use EPOCHREALTIME or date



Suse : use EPOCHREALTIME or date



Debian (dash) : use date



Raspberry-Pi (dash) : use date



ubuntu2604 (dash) : use date



Alpine : use adjtimex and parse the output



FreeBSD : use date or ts



OpenBSD : use ts







--------------------







57 Other alternatives







There are a few alternatives that we haven't discussed yet.







58 Bash with Dash







In the case of Debian, Raspberry Pi, and Ubuntu running dash, since bash is available it is possible to write a separate bash script which simply echos EPOCHREALTIME and then call it from the dash script and capture the output.







While this would work, there's probably not a lot of point to it.



If you can rely on bash being there, then just change the first line of the script and make it a bash script.







59 Adding Packages to Alpine







The ts or timestamp utility is a common unix utility that can be installed if it is not present by default.



This does produce high resolution timestamps on Alpine.



On Alpine Linux this comes as part of the "moreutils" package.



To add the package, use the following



sudo apk add moreutils



echo | ts "%.s"



1779634800.959948







60



You can also add the GNU coreutils, which will provide a high resolution date command which works like in the other examples.



To add the package use the following



sudo apk add coreutils



date '+%s.%N'



1779634800.212897332







61



If you can install more packages into your Alpine system, either of the above two is probably going to be preferable to parsing the output of adjtimex.







62 Custom Timestamp Programs







You can also write a very short program in python, perl, tcl, or some other language and have it output the current epoch time.



I won't discuss that here though.







--------------------







63 Calculating Time Differences







Shell arithmetic is integer only.



If we wish to use high resolution timing data, we need to do something so we don't lose the precision we have worked so hard to get.



There are several possible solutions.







64 Change the Time Base



One method is to change the time base from seconds to milli, micro, or nanoseconds.



This can be done by simply multiplying the time values by the appropriate amount (e.g. 1000, 1,000,000, etc.) before subtracting them.



This allows for integer arithmetic on high resolution values without losing precision.







65 Use the Shell bc Arbitrary Precision Calculator



The bc command line calculator will perform calculations using real numbers and is easy to use in scripts.



It is present by default in most distros.







echo "scale=9; $endtime - $starttime" | bc







where endtime and starttime are variables containing time values.







66



However, for some inexplicable reason, neither Debian nor Opensuse install it by default.



It is present in Ubuntu and Raspberry Pi which are Debian derivatives, and it can be added to distros which lack it.







67 Use awk



awk can also perform calculations using real numbers and it is present in nearly all distros including in all of the ones we tested here.







echo "$endtime $starttime" | awk '{printf "%.6f\n", $1 - $2}'







--------------------







68 Benchmarks







And of course no comparative evaluation would be complete without benchmarks where we see how each method compares to another in terms of speed.







In the benchmark test I ran each method in a loop through multiple iterations, measured the elapsed time, subtracted out the time for an empty loop, and then compared it to alternate methods.



For anything other than EPOCHREALTIME, the empty loop time is negligible and has no real effect on the results.







69



Rather interestingly I came across a bug which caused date to run very slowly if called immediately after using EPOCHREALTIME in bash.



The effect of the bug was to make the date benchmark test roughly 24 times slower.







This has been fixed in newer releases, but if you are using an older distro release then beware of this bug.



I was able to get around it either putting a sleep delay between benchmarking EPOCHREALTIME and benchmarking date, or by simply testing date before testing EPOCHREALTIME.







70



To be able to conduct additional tests I installed ts in Ubuntu and Alpine, and the GNU version of date in Alpine.











71 EPOCHREALTIME Versus date in Ubuntu 2604 bash



The EPOCHREALTIME method is 3103 times faster than date.







However, when the same test is run on Ubuntu 2404 when the date test is run before the EPOCHREALTIME test, EPOCHREALTIME is 1240 faster than date.



Other Linux distros show performance to Ubuntu 2404.



It appears that a side effect of fixing whatever the bug is has the effect of slowing down date.



However, this is probably not a significant issue in normal circumstances.







72 date versus ts in Ubuntu 2604 bash



The date method is 3.7 times faster than ts







73 date versus ts in Ubuntu 2604 dash



The date method is 4.9 times faster than ts







74 date versus ts in Freebsd sh



The date method is 2.5 times faster than ts







75 date versus adjtimex in Alpine Busybox



The date method is 6.0 times faster than adjtimex







76 date versus ts in Alpine Busybox



The date method is 20.0 times faster than ts







77 bc versus awk in Ubuntu 2604



I compared calculating the difference between two numbers when using bc versus awk.



The difference is negligible, with bc being only 7% faster than awk.











78 Conclusion for Benchmarks



Based on these results, if you need to measure elapsed time to high resolution and care about runing the command with as little overhead as possible, then the order of preference should be the following.







79



If you are using a newer version of bash, then use EPOCHREALTIME.



If that is not available, then use date, provided it allows for high resolution times.



If the above two cannot be used, then use ts.



If you are using Busybox and cannot install either GNU date or ts, then use adjtimex.







Date is the closest in terms of being the universal portable solution, but it does not work in all cases.







80



I have not compared different platforms to each other in terms of performance, as that would be a much more involved problem that is outside the scope of this episode.







However, different operating systems implement different commands in different ways.







81



For example, on Openbsd and Freebsd, ts appears to be an ELF binary. That is, it is executable machine code, possibly written in C.



On Ubuntu however, ts appears to be a perl script.



As a result of this, the advantage that date has over ts is much less in Freebsd than it is with Ubuntu (and likely other Linux distros) as on Freebsd it doesn't need to load a perl interpreter to run ts.







--------------------







82 Overall Conclusion



You no doubt thought that measuring elapsed time was going to be so simple, and how could someone get an entire podcast out of such a simple subject?



And yet here we are half an hour later with just a basic overview of the subject.







83



I hope you found this interesting and informative.



Please let us know in the comments if you think that I have done anything incorrectly, or if you have another way of doing things.







I hope to see you all again in another future episode of HPR.







--------------------





Provide feedback on this episode.

This show has been flagged as Clean by the host.

This series is dedicated to exploring little-known—and occasionally useful—trinkets lurking in the dusty corners of UNIX-like operating systems.


In UNIX Curio #8 (
HPR episode 4657
), I talked about using standard utilities to compare files. Left unmentioned, however, was a method commonly used today—the hash function.



As I've stated in previous entries, while I am an engineer, I don't have a background in computer science, so my understanding of the mathematics is limited. But I can give a practical description of what a hash function does. It takes an input, performs a set of calculations on it, and produces an output. As hash functions are practically used, the input is a set of bytes, such as a file or another piece of data like a password. The output is a numerical value in a fixed range—most often, expressed as hexadecimal characters. Because this "hash value" can always be represented in a certain number of bytes, its length as printed is usually a constant number of characters, padded with leading zeros if necessary. This episode will not cover the use of hashes in programming, focusing instead on using them to validate data.



A hash function, or more specifically, a cryptographic hash function, has an additional property. It should be very difficult to predict what changes to the input would be required to produce a specific change in the output.



An older, related concept is called a "checksum". While these are designed to vary when the input data is damaged or digits are transposed, they do not necessarily have that last property mentioned for cryptographic hashes. You have probably already encountered a checksum, even if you didn't recognize it. On a
16-digit number assigned to a Mastercard or Visa


1
credit or debit card, the first six digits identify the card issuer (such as a bank), the next nine digits are assigned to you by the issuer, and the last digit is a check digit. The check digit is calculated using the values of the previous 15 digits, and it is a simple way to avoid typos in entering a card number.



In another example, every
Ethernet frame that your devices send or receive includes a checksum


2
to help ensure that the contents weren't scrambled in transit. This is 32 bits long and is called a cyclical redundancy check, commonly referred to as a CRC. A CRC is also used in many other places—for example, the .zip file format includes one for each archive member, and this allows a program extracting files from the archive to identify if any were damaged.



Our UNIX Curio for today is another example,
the




cksum




utility


3
. It generates a 32-bit CRC based on the Ethernet algorithm. It operates on either a named file or standard input and outputs the CRC value, the length of the input, and the pathname if a file was given as an argument. Unlike most modern hashing programs, the checksum is printed as a decimal integer and is not padded, so it can be anywhere from one to ten digits long. The length value is the number of bytes in the input (actually specified as the number of
octets
, as systems could potentially use a byte that isn't eight bits long), also expressed as a decimal integer.



There are two major ways that one could use
cksum
to check the validity of a file. First, if you are transferring a file from one UNIX-like system to another, you could run
cksum
against it on both systems and check that the CRC and length are the same. The utility can also be given multiple filenames as arguments, which would generate a list that can then be compared. The second way would be for someone publishing a file or set of files to also publish the CRC values, lengths, and names so that people downloading them could verify that they match. However, I don't think the practice of publishing lists like this really started until more recent hash functions like MD5 and SHA-1 came about so it is unlikely that anyone would publish CRC values instead.



The advantage of these tools should be pretty obvious in comparison to
cmp
, one of the utilities discussed in UNIX Curio #8. To verify a file using
cmp
, you need two files to compare—if you're trying to check a large file you downloaded, you would need to spend the time and bandwidth to download a second copy. And if they didn't match, you would have no idea which of the two, if either, was correct. By contrast,
cksum
is quicker to run, doesn't require downloading a massive amount of excess data, and if run against the original file, makes clear what the correct value is.



This utility is a follow-on from a program called
sum
, which operated very much the same. I had a bit of trouble tracking down the exact development history, but what seems clear is that
two different variants


4
were popular: a BSD version and a System V version. Both output 16-bit checksums, but used different algorithms so they didn't give the same results. Also, the BSD version printed the length of the input data as the number of 1,024-byte blocks, while the System V version instead gave a count of 512-byte blocks. (Some sources
claim that System V




sum




generates a 32-bit checksum


5
, which could possibly be true internal to the algorithm, but I have tested several independent implementations of the utility and all of them output a 16-bit value for both the System V and BSD algorithms.)



From what I can tell,
the BSD version


6,7
came first; it was in 3BSD but probably appeared even earlier. An identical
copy of BSD's




sum




was included with UNIX/32V


8,9
, which was AT&T's 1979 port of Seventh Edition UNIX to the VAX and became one of the ancestors of System III. The divergence seems to have started with System III, released in 1980;
its version of the




sum




utility


10,11
changed to a new default algorithm, though it could be made to use the BSD algorithm via the
-r
option. System V looks to have kept the same behavior as System III. It's not clear to me why this algorithm is universally called the "System V algorithm" rather than the "System III algorithm"; perhaps it is because System V saw much more widespread use.



Instead of trying to reconcile these differences, the POSIX committee decided to create a new utility with a unique name, use a separate algorithm entirely, and avoid the block-length dispute by printing the length in octets instead of blocks. I should point out that POSIX states that the CRC algorithm for
cksum
does not strictly meet the mathematical definition of a "checksum". I don't know enough to say exactly
why
it doesn't qualify or to say whether either of the
sum
algorithms do. However, in less-formal usage the term "checksum" has gathered the meaning of any value used to represent or validate a set of data, so I am fine with using it no matter the technical details of the algorithm.



When two different inputs produce the same checksum or hash value, this is called a "collision". Because the output value has a limited range, there are an infinite number of possible inputs that could produce a collision. From a practical standpoint the possibilities are more limited—the majority of these inputs are larger than the number of atoms in the universe, which can't fit on any machine. Unlike a cryptographic hash algorithm, the CRC is not specifically designed to resist an attacker crafting a malicious input that would cause a collision. However, it should be sufficient to detect accidental damage.



Programs implementing more modern cryptographic hash algorithms are superior to the checksum utilities in avoiding collisions (whether malicious or accidental), but there are still three advantages that the older programs have. First, a system running a historical operating system might not have the hash programs available, but is more likely to have
cksum
or
sum
already included. Second, the checksum values are much shorter than the hashes output by the newer programs, making them easier for a user to compare by looking at them. This advantage is not as great as it might appear at first, because a common way to check a hash these days is to save a list of hashes and filenames—the hash programs can use that and do the comparison themselves, sparing the user from having to validate it character by character. The third advantage is that
cksum
prints the input length in bytes. This greatly limits the number of inputs that could be maliciously crafted to create a collision.



I did a moderate amount of research on implementations of modern cryptographic hash algorithms and found that some, such as MD5, SHA-1, and SHA-2, do use the length of the input (often termed "message length" in the literature) as part of the material fed in to the algorithm, but none of the hashing utilities present this length to the user as part of its output. There are two possible reasons for this that seem evident to me. First, if one is hashing a password, you would certainly not want to give a clear indication of its length—that would give any attacker a massive head start on guessing the password. However, that doesn't explain why one would avoid printing the input length for a file that is made publicly available. Second, it is convenient in many contexts, such as database entries or in software (such as
git
), for the hash to be a fixed length. Including an extra value that can be of variable length would complicate those use cases. However, the length value could simply be dropped and they would be no worse off than they are currently.



Historically on UNIX, password hashing was treated differently from checksumming files—
the




crypt()




function


12
was used for passwords while
sum
and later
cksum
were used to confirm a file's integrity. So even rather early on, these two use cases employed algorithms with different properties, but I haven't dived into the history deeply enough to know how intentional this was. My discussion in this episode focuses on the file use case, so understand that I'm largely avoiding the topic of password hashing. Digital signatures are yet another use case, one that I'm ignoring entirely.



Every few years, some security researcher declares a particular hash algorithm to be "broken" and that everyone should move over to a new one, which generally has a longer hash. While the larger hash space certainly reduces the opportunity for collisions, this disrupts workflows, such as publishing information about software releases by e-mail, which still tends to observe a
78-character limit on each line


13
, making it harder to include a list of hashes with filenames next to them. This is in addition to the work of modifying software and scripts to use the new algorithm and managing how to treat past data. It seems to me that publishing the input length along with the hash would make it far more difficult to craft a malicious input that matches both, but I haven't found discussion of that during my investigation. (See the Appendix for a possible implementation.) Perhaps someone listening can record a response episode for HPR explaining that.



References:







Payment card number
https://en.wikipedia.org/wiki/Payment_card_number





Ethernet frame: Frame check sequence
https://en.wikipedia.org/wiki/Ethernet_frame#Frame_check_sequence





Cksum specification
https://pubs.opengroup.org/onlinepubs/009695399/utilities/cksum.html





GNU coreutils manual: sum
https://www.gnu.org/software/coreutils/manual/html_node/sum-invocation.html





FreeBSD 15.0 sum manual page
https://man.freebsd.org/cgi/man.cgi?query=sum&sektion=1&manpath=FreeBSD+15.0-RELEASE+and+Ports





3BSD sum manual page
https://www.tuhs.org/cgi-bin/utree.pl?file=3BSD/usr/man/man1/sum.1





3BSD sum source
https://www.tuhs.org/cgi-bin/utree.pl?file=3BSD/usr/src/cmd/sum.c





UNIX/32V sum manual page
https://www.tuhs.org/cgi-bin/utree.pl?file=32V/usr/man/man1/sum.1





UNIX/32V sum source
https://www.tuhs.org/cgi-bin/utree.pl?file=32V/usr/src/cmd/sum.c





System III sum manual page
https://www.tuhs.org/cgi-bin/utree.pl?file=SysIII/usr/src/man/man1/sum.1





System III sum source
https://www.tuhs.org/cgi-bin/utree.pl?file=SysIII/usr/src/cmd/sum.c





Crypt specification
https://pubs.opengroup.org/onlinepubs/009695399/functions/crypt.html





RFC 2822: Internet Message Format: Line Length Limits
https://datatracker.ietf.org/doc/html/rfc2822#section-2.1.1





OpenSSH 10.1 released
https://lwn.net/ml/all/dd12623ae86aa5eb@cvs.openbsd.org/







Appendix




The MD5 hash algorithm was (and still is) widely used, but many people characterize it as being "broken" and discourage its use. Let us imagine a variant of this, called MD5.L, where the normal MD5 hash is followed by a "." character and the input length expressed as a hexadecimal number.



Take, for example, the
e-mail message announcing the release of OpenSSH 10.1


14
. At the bottom, it includes an SHA-1 hash and an SHA-2 256-bit hash for the available gzipped
tar
files. That longer hash is encoded with Base64 because if it were given as a hexadecimal number, it would make the line longer than 78 bytes. The MD5.L hash of the file would be one character shorter than the SHA-1 hash, as shown below. (The extra length of the
name
makes them both consume the same number of characters. The hashes shown are for the "portable" version of OpenSSH.)



Some people claim SHA-1 is also broken, seeking to have people use newer and longer hash functions. For an attacker to compromise MD5.L in this example, they would not only have to create a valid
tar
file compressed with
gzip
containing a malicious payload having the right MD5 hash, that file would have to be exactly 1,972,831 bytes long (the decimal equivalent of 1e1a5f). While there are still many possible inputs that could be tried (256
1972831
, to be exact*), this is far fewer than the infinite possibilities for plain MD5, SHA-1, or SHA-2.



If for some reason it is super important to have a fixed hash length, let's imagine another variation called MD5+L. In this one, instead of L being the input length, it is the input length
modulo
one terabyte (2
40
bytes), which can be represented by 10 hexadecimal characters, left-padded with zeros. While this approach substantially increases the number of possible inputs an attacker could try, it is likely that an intended victim would notice that the file they downloaded is larger (or smaller) than expected by that much. The MD5+L hash is longer than a SHA-1 hash, but still shorter than a 256-bit SHA-2 hash.



SHA1 (openssh-10.1p1.tar.gz) = 7fd17b99d1beffb47cd380d64079e920bb0bd91f
SHA256 (openssh-10.1p1.tar.gz) = ufx6K4JXlGem8vQ+SoHI4d/aYU3bT5slWq/XAgu/B1g=
MD5.L (openssh-10.1p1.tar.gz) = 80dd9bb00a86519934710d05903fdf07.1e1a5f
MD5+L (openssh-10.1p1.tar.gz) = 80dd9bb00a86519934710d05903fdf07+00001e1a5f



Of course, if MD5 is considered to be too weak even with the inclusion of the length, one could produce a ".L" or "+L" version of any hash function. However, longer hashes
will
end up running into the 78-character limit.





*This is a number with 4.75 million digits that the




bc




utility on my laptop took almost 5 minutes to calculate.


Provide feedback on this episode.

This show has been flagged as Explicit by the host.

New hosts
Welcome to our new hosts:
Lennart Benschop

Last Month's Shows
Id Day Date Title Host 4651 Mon 2026-06-01 HPR Community News for May 2026 HPR Volunteers 4652 Tue 2026-06-02 simon says Brian-in-Ohio 4653 Wed 2026-06-03 Starting the Habit of Reading Thaj Sara 4654 Thu 2026-06-04 What's in my component Box? MrX 4655 Fri 2026-06-05 Tips for Glasses norrist 4656 Mon 2026-06-08 My review of musicozy sleep/exercise bluetooth headband Swift110 4657 Tue 2026-06-09 UNIX Curio #8 - Comparing Files Vance 4658 Wed 2026-06-10 Audio Revisited Whiskeyjack 4659 Thu 2026-06-11 Command Line Fun - Recording a show Kevie 4660 Fri 2026-06-12 Robert A. Heinlein: The Future History, Part 1 Ahuka 4661 Mon 2026-06-15 Laptop Computer Woes, or How I Learned to Love My Tech Hoarding Claudio Miranda 4662 Tue 2026-06-16 “What Are the Answers I Need, To the Questions I Don't Know Enough to Ask?” Antoine 4663 Wed 2026-06-17 The hallway track at T-DOSE Ken Fallon 4664 Thu 2026-06-18 No Input Mixing Tutorial and How to Build a Drone Box TheDUDE 4665 Fri 2026-06-19 Pokémon GO Lee 4666 Mon 2026-06-22 How I got into tech Lennart Benschop 4667 Tue 2026-06-23 UNIX Curio #9 - printf Vance 4668 Wed 2026-06-24 Nuclear Power Technology Follow Up on Safety Whiskeyjack 4669 Thu 2026-06-25 HPR Beer Garden 14 - Super Strong Lager Kevie 4670 Fri 2026-06-26 Playing Civilization V, Part 13 Ahuka 4671 Mon 2026-06-29 Protocal AI operat0r 4672 Tue 2026-06-30 Hey Mum, I'm on Spotify ! Ken Fallon Comments this month
Past shows
hpr4181 (2024-08-12) "Downloading out of copyright movies" by Bob.

Jan said: "subtitles" (2026-06-29 21:55:12)

hpr4633 (2026-05-06) "Ham Radio Licence" by Lee.

RJ said: "Very interesting as always" (2026-06-02 11:22:47)

Lee said: "Aerials" (2026-06-02 16:44:45)

hpr4644 (2026-05-21) "Response to comments on HPR4424: Newsboat..." by Archer72.

Archer72 said: "Not quite a complete script" (2026-06-07 22:26:34)

Whiskeyjack said: "Response to hpr4644" (2026-06-08 18:01:33)

Archer72 said: "Response to WhiskeyJack" (2026-06-12 16:50:42)

Whiskeyjack said: "Response to Archer72 on HPR4644" (2026-06-13 11:13:49)

Archer72 said: "EyeD3" (2026-06-13 21:10:48)

hpr4649 (2026-05-28) "What did I do at work today? Part 3 Section 2" by Lee.

Ken Fallon said: "Love PHP" (2026-06-02 10:52:54)

candycanearter07 said: "Re: Love PHP" (2026-06-02 17:24:01)

hpr4650 (2026-05-29) "Playing Civilization V, Part 12" by Ahuka.

Antoine said: "Puppetting and Happiness" (2026-05-29 14:21:14)

Kevin O'Brien said: "That's the algorithm" (2026-06-01 21:13:58)

Antoine said: "#2 Thanks!" (2026-06-04 23:25:42)

This month's shows
hpr4651 (2026-06-01) "HPR Community News for May 2026" by HPR Volunteers.

candycanearter07 said: "busy weeknd" (2026-06-02 02:38:55)

hpr4653 (2026-06-03) "Starting the Habit of Reading" by Thaj Sara.

Antoine said: "Reading always has been a life-saver for me, this alone makes this show Very Nice" (2026-06-04 23:37:27)

Kevie said: "Nice episode" (2026-06-10 08:11:43)

hpr4654 (2026-06-04) "What's in my component Box?" by MrX.

Kevie said: "Geeks are ahead of their time" (2026-06-10 08:14:12)

hpr4655 (2026-06-05) "Tips for Glasses" by norrist.

Trey said: "Avoid hand soap " (2026-06-05 12:11:33)

Jim DeVore said: "Geeks are lazy, too" (2026-06-21 00:10:45)

Operat0r said: "Glasses" (2026-06-26 18:53:44)

hpr4656 (2026-06-08) "My review of musicozy sleep/exercise bluetooth headband" by Swift110.

candycanearter07 said: "wishlist!" (2026-06-08 13:12:08)

Kevie said: "Interesting" (2026-06-10 08:15:59)

hpr4657 (2026-06-09) "UNIX Curio #8 - Comparing Files" by Vance.

xmanmonk said: "Great Show (again)" (2026-06-09 23:31:58)

candycanearter07 said: "comparisons" (2026-06-11 16:57:57)

Whiskeyjack said: "HPR4657 - use of comm" (2026-06-13 11:47:53)

Vance said: "Appreciate the comments" (2026-06-15 03:16:05)

Whiskeyjack said: "Reply to Vance on awk in HPR4657" (2026-06-15 17:02:57)

hpr4659 (2026-06-11) "Command Line Fun - Recording a show" by Kevie.

candycanearter07 said: "ffmpeg concatenation" (2026-06-15 11:58:49)

Whiskeyjack said: "Reply to candycanearter07 on HPR4659" (2026-06-15 16:53:16)

hpr4661 (2026-06-15) "Laptop Computer Woes, or How I Learned to Love My Tech Hoarding" by Claudio Miranda.

candycanearter07 said: "uses :)" (2026-06-16 15:04:20)

xmanmonk said: "Great show!" (2026-06-17 04:37:48)

ClaudioM said: "Thanks! (In reply to candycanearter & xmanmonk)" (2026-06-17 14:02:17)

hpr4666 (2026-06-22) "How I got into tech" by Lennart Benschop.

Windigo said: "Nice to meet you" (2026-06-26 02:24:48)

candycanearter07 said: "hello!" (2026-06-27 15:04:02)

hpr4667 (2026-06-23) "UNIX Curio #9 - printf" by Vance.

candycanearter07 said: "learned something new :)" (2026-06-23 13:23:06)

Whiskeyjack said: "HPR4667 - printf" (2026-06-24 03:54:20)

xmanmonk said: "Another great show!" (2026-06-24 21:15:22)

Mailing List discussions
Policy decisions surrounding HPR are taken by the community as a whole. This discussion takes place on the Mailing List which is open to all HPR listeners and contributors. The discussions are open and available on the HPR server under Mailman.
The threaded discussions this month can be found here:
https://lists.hackerpublicradio.com/pipermail/hpr/2026-June/thread.html Events Calendar
With the kind permission of LWN.net we are linking to The LWN.net Community Calendar.
Quoting the site:
This is the LWN.net community event calendar, where we track events of interest to people using and developing Linux and free software. Clicking on individual events will take you to the appropriate web page. Provide feedback on this episode.