Hacker Public Radio

This show has been flagged as Clean by the host.

--------------------







01 Introduction







This is the fifth episode in an 8 part series.







02



In the previous two episodes we looked at the Allen Bradley PLC 2 and the Siemens S5 series of PLCs.



In those episodes we mentioned I/O modules, but didn't really go into any detail on them.



Instead we looked at the CPU units and focused on things from the programmer's perspective.







03



In this episode we will look at the I/O modules, what they are, how they work, and the purpose that they serve in the system.







In addition to this, I will discuss machine safety systems and also claims made by certain people about cyber security and industrial safety.







--------------------







04 What is I/O?







First though, we must address the question of what I/O is.



While the CPU module may run the program, in order for the program to do anything useful it must interact with the real world outside the PLC.



It must sense the state of the machine it is controlling through inputs, and it must affect the operation of the machine through outputs.







05 A Simple Example



Let's take a very simple example to try to make the above a bit more tangible.



Assume that we wish to press two parts together, such as pressing a bearing into a housing.



This is a very common industrial process.







06



Let us assume we have the following items which connect to the inputs.



* An emergency stop circuit.



* A safety system to protect the operator.



* A part presence sensor used to detect that the housing was correctly placed in the tooling.



* Another part presence sensor used to detect that the bearing was correctly placed in the tooling.







07



* A sensor which indicates the hydraulic cylinder is fully retracted.



* Another sensor which indicates the hydraulic cylinder is fully extended and the part was fully pressed home.



* A button which the operator uses to initiate the press cycle.



* Another button which the operator can use to abort the press cycle in the event of a fault.







08



We also have the following items which connect to the outputs



* A hydraulic valve with two solenoids (down and up).



* A pilot light which tells the operator that the cycle was completed successfully and the part is good.



* A pilot light which tells the operator that the cycle was not completed successfully and the part is bad.







This gives us a total of 8 inputs and 4 outputs.







09



A typical very small PLC would be suitable for this application.



The PLC program would look at the state of the inputs, and apply the necessary logic to turn the outputs on or off at the appropriate time.







As you can see, what makes a PLC useful is its ability to control I/O.







10



Small PLCs typically have a limited amount of I/O built into the same package as the CPU and can be used as is right out of the box.







Medium to large PLCs will interface to racks of I/O, and the number of I/O points can be in the dozens to hundreds and even the thousands.







--------------------







11 Types of I/O



The above example is very simple.



There are in fact many types of I/O.



I will outline the most common ones here.







12 Digital I/O



The first is "digital" I/O.



This is overwhelmingly the most common type.



Digital I/O are basic on-off signals.



A digital input will detect if a switch is on or off.



A digital output will turn a relay or valve on or off.







13



Digital I/O comes in three main types categorized by the type of electronic device used.



These are



* AC



* Relay



* DC







14 AC



AC I/O was the earliest type, but has largely faded away by now.



This works with 100 to120 VAC or 230 VAC.



Inputs detect the presence or absence of these voltages, and outputs output these voltages when turned on.



The relay logic that PLCs replaced was largely driven by these voltages, so being able to interface with existing systems was an important design consideration for early PLCs.







15 Relay



Relay outputs were an alternative to triac devices used for solid state AC outputs.



These were very small relays built into the output modules.



Their main attraction was they were usually somewhat cheaper than solid state outputs.



However, they suffered from being much less durable and had a shorter life.







16 DC



By the 1980s and 1990s, AC and relay I/O had been largely replaced in new applications by 24VDC.



This allowed for more compact and less expensive control wiring, and fewer electrical safety concerns.



Both inputs and outputs worked with 24VDC.



All of the standard sensors, solenoid valves, relays, push buttons, lights, and pretty much all other devices were designed to be compatible with 24VDC.



In cases where you had large motor contactors or relays, or large solenoid valves which required higher voltages for practical reasons, an additional interposing relay could be used to amplify the control signal.







17



I/O modules or cards came in 8, 16, 32 point sizes, and there were modules which combined both inputs and outputs in a single module.







--------------------







18 Analogue I/O



As well as simple on-off voltages, PLCs could also work with variable voltages, where the actual voltage or current represents a measurable value, such as temperature or dimension.



These are known as analogue values.



These can be inputs or outputs and are known as analogue inputs or analogue outputs.







19



The simplest is voltage.



The most common standard voltage ranges are



* 0 to 10V.



* Plus or minus 10V.



* 0 to 5V.



* 1 to 5V.







20



Another type is current.



This is where the amount of current rather than the level of the voltage is what is significant.



Current has an advantage over voltage in that it is less affected by resistance losses in long wiring runs and is also less prone to being influenced by electrical noise due to the lower impedance.







21



The most common standard current ranges are



4 to 20 mA.



0 to 20 mA.







22



Another common type is thermocouple inputs.



These have special circuitry built into them to interface with thermocouple devices, which are a common device used to measure temperature by measuring the voltage generated when two dissimilar metals are heated in contact with each other.











--------------------







23 Serial Communications



A number of PLCs have serial communications ports, mainly used to interface with things like label printers or bar code readers.



Some of these have even had basic interpreters built into them, mainly to allow for better string formatting or parsing.







24 Servo and Stepper



Another category of I/O are servo and stepper motor controllers.











25 High Speed Counter Timer



Another type are counter/timer modules, which allow for high speed counting of events, or very precise timing of events in hardware, or interfacing with rotary or linear encoders.



Encoders are measuring devices which have a series of very small but precise marks which interrupt beams of light when they move, allowing measurements to be made.







26 Networking



Many PLCs also support some sort of networking.



I won't go into any detail on that here, but will instead address this area in another episode.



However, network ports may be built into the CPU, or may reside in separate network I/O modules.







27 Specialty



As well as the preceding there are many other specialty modules which I won't bother to describe here.







--------------------







28 Operator Panels



The original operator panels were arrays of push buttons, pilot lights, and other basic operator controls mounted in boxes for an operator to interact with the machine.







Later on these were replaced by specialty electronic devices which were also known as operator panels as they served the same function.







29



Modern operator panels are screens and keypads which are used to display information to an operator or to allow data to be entered into the PLC.



While not strictly speaking a type of I/O, operator panels typically look like I/O to the PLC CPU.







Buttons on a touch screen may be effectively the same thing as actual hardware buttons from the PLC's perspective, and analogue values output to the operator panel for display are like hardware analogue devices.







30



All of these data are mapped to memory locations in the PLC, with the details of how this is done varying according to the make and model of PLC.







31



I won't go into any detail on this as this could be a separate subject on its own.



Generally though, you would use software to create screens to display on the operator panel.



The indicators and buttons defined in your screens would be mapped to addresses in the PLC, either directly or indirectly.



This configuration would then be downloaded into the operator panel.



Your PLC program could then interact with the operator panel using ordinary ladder logic.











--------------------







32 Safety Systems



Up to now I have mentioned only PLCs in any detail.



There is however more to a complete machine control system than just a PLC.



One of these is safety systems.







33



I will leave out historical safety systems and just focus on modern ones.



I will also leave out large process industries involving things like boilers or chemical reactors and just focus on assembly operations in industries such as automobiles, appliances, and other similar things.



This won't be a tutorial on safety design, just a very brief overview.







34



An industrial machine must be designed with safety as one of its fundamental design principles.



If the machine cannot be operated safely, then it cannot be put into use.



Safety must take into account the operators, bystanders, and maintainers.



You must assume that people will be inattentive or make mistakes.







35



You must also assume that the PLC program will have bugs and that the electronics will fail in a state which could cause a hazard if it is permitted to.



In other words, the PLC and your PLC program are not part of the safety system.







36



The safety system will decide when the PLC will be permitted to act and when it will not.



The PLC in turn must monitor the safety system in order to understand when it can and cannot act.



Your PLC program must take this into account so that it can differentiate between not being able to act because the safety system has decided it is not safe to cycle, versus there being something wrong with the machine which prevents it from operating.







37 A Few Basic Safety Devices



Here are a few basic safety devices.



I will not attempt to be comprehensive.



Rather, I am just giving a few common examples so that you will understand the environment in which the PLC can operate.







38 Emergency Stop Circuit



A basic element of nearly all industrial machines is some sort of emergency stop circuit.



When the "on" button is pressed, a relay will close and hold itself on.



This will enable power to the machine, including the compressed air supply if there is one.



If the "stop" or "e-stop" button is pressed, the relay will open and remove energy from the machine, rendering it safe.



If the electrical power supply to the machine goes off, due to for example the building losing electrical power, the emergency stop circuit will also "drop out" or open, and remain off until the start button is pressed again.







39



While every, or nearly every, machine will have an emergency stop circuit, is it often not sufficient on its own.



In other words, it is necessary, but not always sufficient.







40 Guard Switches



Equipment hazards must be enclosed by protective guarding.



Movable guards, such as access doors, must be interlocked with the safety circuit such that when the guard is opened, the machine is rendered safe.



To achieve this, there will be safety rated switches on each access door which are connected to the safety circuits of the machine.







41 Two Hand Control



Another common safety circuit is two hand control, often called "anti tie down", although that latter phrase actually describes one of the design principles rather than being the proper name.



With two hand control, the operator station will have two large buttons which the operator must press simultaneously and hold to enable the machine to operate.







42



With both hands on the buttons, you know where the operator's hands are.



The operator must remove both hands from the buttons in order to enable the circuit again.



If one button remains closed and the operator attempts to activate the machine again with the other button, then the safety system will detect this and refuse to enable the machine again until both buttons are de-activated.







43



This is the "anti tie down" feature.



The operator cannot tie one button down and cycle the machine repeatedly with just the other button.



The buttons will be connected to a special relay which monitors and self tests to ensure it operates correctly.







44



The PLC will monitor an auxiliary contact on this relay so that it knows when to start its sequence of operation, but the redundant safety contacts on the relay lie in between the PLC and whatever hazards are being guarded against.



The safety device has ultimate control over whether the machine moves or not.







45 Light Curtain



Another basic safety device is the light curtain.



This has a series of closely spaced light beams across the opening which forms the operator's access to the working area of the machine.







46



If any of these light beams is interrupted, the light curtain controller will not permit the machine to move.



Light curtains can usually be used in places where two hand control systems can be used.



They have an advantage over two hand control in that the operator can be performing other tasks while the machine is cycling instead of being occupied with holding the two hand control buttons down.



This allows for productivity improvements which can usually more than pay for the higher cost of the light curtain compared to simple two hand control.







47 Other Devices



There are many other safety devices, many of which address niche applications.



I won't bother listing any more than I have so far however, as you should understand the basic principles by now.







48 Safety PLCs



There are special safety PLCs which have redundant CPUs which monitor one another, and which have redundant mutually interlocked I/O, and which run safety reviewed software.



Anything connected to a safety PLC must itself be safety rated if it could affect the safety of the machine.



However, these are quite rare and are only used in certain special industries.







49



These are normally used in places where some programmed sequence of operations is required to render a system safe.



This is more commonly found in places such as oil refineries, chemical plants, or electric power generating plants than in typical factories.







--------------------







50 Safety and Cyber Security



This is a good place to bring up the question of safety and cyber security.



There have been a number of articles in the IT press recently about how PLCs could be indirectly affected by computer viruses and how this could cause dangerous safety problems in manufacturing equipment.







51



The premise is that they found security holes in software which runs on personal computers running Windows operating systems which are used to do things like record production data.



There are a number of software systems which are designed for this purpose, although SCADA systems are usually the ones mentioned.







52



I will briefly describe SCADA systems in a later episode.



For our purposes here however, just understand it as specialized industrial software that runs on a PC and conducts monitoring and display using data from a PLC that it is networked to.







53



According to these security researchers, a virus could exploit these security holes to send commands to the PLC and cause it to do arbitrary things which could cause injury or death.



The PLCs being referred to in this instance are ordinary PLCs, not safety PLCs.







54



I find these claims to be highly implausible for the reasons that I have stated above when describing safety systems.



An ordinary PLC is not a safety device, and so the safety of the machine should not be affected by a malfunctioning PLC, regardless of whether any viruses were involved or not.







55



I have seen PLCs malfunction due to memory corruption errors, but these posed loss of production problems, not safety problems.



A malfunctioning PLC may cause property damage to the machine and loss of money, but the control system, of which the PLC is only a part, should not permit it to pose a hazard to personnel.







56



I have not seen any of these security researchers offer any plausible mechanism by which their claims of safety problems may be realized.



As a result of this, I take all such claims with a very, very, very, large grain of salt.







--------------------







57 Conclusion







In this episode we looked at the I/O modules, what they are, how they work, and the purpose that they serve in the system.







In addition to this, we discussed machine safety systems and also claims made by certain people about cyber security and industrial safety.







58



In the next episode I will cover networking, including a detailed look at one industrial network protocol, called Modbus.







59



This has been the fifth episode in an 8 part series.







--------------------





Provide feedback on this episode.

This show has been flagged as Clean by the host.

This series is dedicated to exploring little-known—and occasionally useful—trinkets lurking in the dusty corners of UNIX-like operating systems.


Thirteen is one of my favorite numbers, so I am going to use this episode to indulge myself. As part of UNIX Curio #8 (
HPR episode 4657
), I went on a little flight of fancy and suggested that the
comm
utility could be used in place of
cat
. On further reflection, I realized that there are quite a few utilities that can do that job, almost. This episode is the result of me investigating how many possible ways there are to replace that functionality with standard utilities. Imagine that some prankster has deleted the
cat
utility from your machine, and you need a way to restore that capability without re-installing it.



I should warn you that as opposed to the rest of the series, this episode will not focus heavily on obscure tools, and it is unlikely the examples given will be very useful. Also, I am not going to dive into the history of any of these commands. However, you might still learn something, and it could be a little entertaining to hear about the ways some utilities can be twisted to this purpose.



Let's start by reviewing what the
functionality of




cat


[1] is. Under POSIX, all it does is read the data provided on standard input or from each file named as an argument, and writes it to standard output. Briefly, it con
cat
enates all of the entries. The only option required by the standard is
-u
, which causes
cat
to write data without any delay (if that option were not given, the program could choose to buffer the data). While many implementations offer additional options that modify the output, I am going to completely ignore them and just focus on that one job. The files named as arguments are processed in the order in which they are named. One or more of these can be
-
, in which case
cat
will use the data provided on standard input and then continue to the next file in the argument list. I will actually narrow things even further and allow discussion of cases where only one file is involved, or where
cat
is reading from standard input via redirection from a file.



I will start off with the
TL;DR
conclusion—none of the standard utilities are able to fully duplicate its functionality on their own. Some of them come close, but can't quite be relied on in all respects. One important reason for this lies with the
POSIX definition of a text file
[2]. This is a file treated as being separated into lines, divided by newline characters. There are three limitations: none of the lines can contain a NUL byte (value 0), the lines cannot exceed the maximum line length defined by the system (required to be at least 4096 bytes including the trailing newline), and the file must end with a newline character. (The last rule is implied by the
definition of "line"
, which states a newline must appear at the end of a line.) It might seem strange, but a text file
may
contain non-ASCII data; the only restriction on the values of bytes is that they cannot be NUL. Many of the standard utilities are only required to operate on text files and therefore cannot necessarily be relied on to handle arbitrary data. While a particular implementation might not have these limitations, there is no guarantee of that. I should note that both
cut
and
paste
are only required by POSIX to operate on text files, but with the exception that the lines can be of any length.



In addition to
cat
, the only utilities covered in this episode that are required by POSIX to handle any arbitrary file are
dd
,
tail
(when used with the
-c
option),
tee
,
tr
, and
uuencode
. The wording for the
compress
utility does not specifically
say
it must handle any file, but in practice I think this is true. However, all of these utilities have at least one limitation of their own: they cannot handle more than one file or they can only operate on standard input. Some other utilities are able to deal with multiple files, standard input, or both, but those are only required to operate on text files. In practice, this typically means that if an input file does not end with a newline character, the program will add one to the output, destroying the idea that the output of
cat
should be identical to the input.



Let's start with the least-capable options and work our way towards those that could serve as closer drop-in replacements. The first candidate is a utility that I thought might be persuaded to work, but it doesn't quite. This is


nl




, a command used to number lines of files or standard input
[3]. It normally prints the content of this input and, on the left, prints the number of each line. It
is
possible to turn off the numbering, and to set the separator between the numbers and the text to a null string. However, it does not seem to be possible to have absolutely nothing appear in front of each line. The POSIX specification states that the
-w
option can be given to define how many characters to the left of each line will be reserved for the line numbers—while the default value is six, it says nothing about what values must be accepted. I tested several implementations (specifically, from GNU, FreeBSD, NetBSD, and OpenIndiana, which is an open-source derivative of System V) and none of them allow setting the width of the space for line numbers to be less than one. Even if it were possible to remove the preceding space entirely, POSIX only requires
nl
to accept one file argument and to operate on text files, so it would still be limited.





In the examples for this episode,


myfile


is used to represent the name of the file being operated on. The text


myfile ...


means a filename possibly followed by other filenames (with no limitation on the number). The




nl




example below prints a single space in front of each line.




nl -b n -s "" -w 1 myfile



Next up are methods that actually work, but are only usable with standard input. When combined with input redirection from a file, they can accept a single file and send it to standard output.
The




read




utility
[4] takes a line of input and assigns it to a shell variable. It requires a little help, however—it can't output what it reads, so it has to be paired with something like
echo
or
printf
. Because it reads input one line at a time, it needs to be used in a loop to read the entirety of the data. In addition, the
IFS
shell variable must be set to a null string to prevent
read
from doing field splitting; we don't want it to modify the data being read at all. The shell loop
while IFS="" read -r foo; do echo "$foo"; done < myfile
will do the job, except that
read
is only required to handle text files. It is not clear to me whether POSIX allows the shell to have a limit on how much data can be stuffed into a variable; if it does, that could be a problem even if your implementation of
read
can handle lines longer than the maximum length.





The




tr




utility
[5] is another one that only works with standard input. It takes the input, replaces any characters in the first string argument with the corresponding characters in the second string argument, and sends the result to standard output. If the first and second strings are the same, such as
tr a a < myfile
, it will behave like
cat
.





The




tee




command
[6] also fits into this category. It copies standard input to standard output, just like
cat
does. It also sends a copy to any filename arguments given. If none of these are present, it just passes the data through untouched. You could use
tee < myfile
to print a file. Be careful, however, not to enter your filename as an argument (as in
tee myfile
), as that will cause it to be overwritten! The good news with both
tr
and
tee
is that they aren't limited to working with text files.



Let's now consider a couple of options that only work on a file, not with standard input. In addition, they can only handle
one single
file. This might seem to defeat the purpose of
cat
, which can combine multiple files, but
cat
is often used with only one file. These are


ed


[7] and


ex


[8], two text editors. By feeding them the right command sequence on standard input, they can be convinced to output the entire file and then exit. The appropriate invocations would be
printf '%s\n' '1,$p' Q | ed -s myfile
and
printf '%s\n' '1,$p' 'q!' | ex -s myfile
. The POSIX specification for
ex
indicates that more than one filename can be given as an argument. I attempted to take advantage of this with the command line
while true ; do printf '%s\n' '1,$p' 'n!' ; done | ex -s myfile ...
, which should print the first file named as an argument to standard output and move to the next one. In my reading of the specification, once the last file has been printed, the
n!
command (meaning "next") should result in an error, causing the program to exit. However, the implementations that I tested don't exit and just stay on the last file, printing it out endlessly. Perhaps I'm misunderstanding how
ex
works, but regardless, both it and
ed
are only required to handle text files, so can't completely serve in place of
cat
anyway.



The following category of replacements is capable of using either standard input or a file, but they can only work with one file. Let's start with


comm


[9], which was responsible for me exploring this question. The command
comm myfile /dev/null
will send
myfile
to standard output. If you want to use standard input instead of a file, you need to replace the first filename with a hyphen (as in
comm - /dev/null < myfile
). The


paste




utility
[10] works similarly. It can only be used with one file because if more are given, it will try to interleave the lines of these files together. Another possibility is


tail


[11]. When used with the
-c +1
option, it outputs the entirety of the file. Unlike other implementations I tried, the NetBSD version of
tail

is
capable of taking multiple filenames as arguments. However, in that case, it precedes the content of each file with the filename like
head
does, so doesn't produce a clean copy.
The




sort




utility
[12] used with the
-m
option, which I talked about in UNIX Curio #11 (
HPR 4687
), can also do the job. All of these are only required to handle text files by POSIX except
tail
, which must handle any arbitrary file when the
-c
option is used.



One more utility that can take either one file or standard input is


dd


[13]. It is a bit different from these others in a couple ways. First, it writes statistics about the data processed to standard error, so you would need to redirect that to
/dev/null
to replicate the behavior of
cat
. To use standard input, no arguments are needed. The syntax for arguments is unusual compared to other UNIX utilities; for this purpose,
dd if=myfile 2>/dev/null
would be the command to use.



The


compress


[14] and


uuencode


[15] utilities are in the same category, but do a lot more processing on the data along the way. The purpose of these is described respectively in UNIX Curios #7 (
HPR 4647
) and #1 (
HPR 4587
). Their output can be piped into a corresponding
uncompress
or
uudecode
utility to get back the original data and send it to standard output. The two commands would be
compress -c myfile | uncompress -c
and
uuencode myfile /dev/stdout | uudecode
—if you wanted to use standard input instead, just omit the filename arguments. I should point out that when I tested the
uuencode
method on FreeBSD 15.0, it didn't work; however, it did work on NetBSD 10.1 and OpenIndiana 2025.10.



Next, we have two possibilities that come very close, but don't quite behave
exactly
like
cat
. They send to standard output either the files named as arguments in order, or, if there are none, standard input. However, POSIX does not require them to take a hyphen as an argument, representing standard input. First up, we have


sed


[16]; when called with an empty script (as in
sed "" myfile ...
or
sed :a myfile ...
), it just prints out the contents without any transformations.
The




grep




utility
[17] works the same way—when given an empty regular expression (as in
grep "" myfile ...
), it treats that as matching anything and prints each line of input to standard output.



Finally, we have arrived at the utilities that can fully replace
cat
, albeit only for text files. They can be given one or more files, including a hyphen representing standard input, and will output them in the order they are given as arguments. If no filename arguments are given, they just use standard input instead. Starting out, we have


awk


[18]. While an empty pattern matches every line of input and an empty action prints the entire line, POSIX states that if both the pattern
and
action are empty, the utility is to simply exit without reading or writing anything. Therefore, we need to at least provide a pattern or an action—a pattern, as in
awk "/.*/" myfile ...
, can be shorter. Another one that has the capabilities is


cut -b 1- myfile ...


[19]. However, as noted earlier, these will add a newline character if the input doesn't already end with one, so they aren't exact replacements.



I was hoping that a simple invocation of one of the standard utilities could be used in place of the
cat
program that our prankster has deleted; however, I think I've demonstrated that isn't possible. Let's return to those utilities that can handle arbitrary files, but can't take more than one at a time. It
would
be possible to write a shell script to manage any options and arguments, then use the utility to do the actual work of sending through the data file by file. In my view,
tail
would be the best fit because it can take a bare filename as an argument and, unlike
dd
, uses the standard syntax for UNIX utilities.



The following script does just that—unfortunately, it is not really something simple that could be typed in from memory. So probably it would be easier in most situations to just re-install
cat
on your system instead of looking up this script, typing it in, and saving it where
cat
used to be.



#!/bin/sh
# Exit if any option other than -u is given; 'getopts' will issue an error
# message. We don't actually do anything different if -u is given.
while getopts u catopt
do
if [ "X$catopt" != "Xu" ]
then
exit 1
fi
done

# Reset the positional parameters to remove any options given.
shift $(($OPTIND - 1))

case $# in
0)
tail -c +1
;;
*)
for fname
do
case "$fname" in
-)
# Single hyphen represents standard input, same as with no arguments.
tail -c +1
;;
*)
# The -- ensures that $fname will be treated by 'tail' as an argument,
# not an option.
tail -c +1 -- "$fname"
esac
done
esac



It might be tempting to implement this substitute as an alias or a shell function rather than as a script, but those only exist in the current shell execution environment (and its subshells) and are not inherited by utilities and scripts run by you or others. So the easiest way to get it to be used by other scripts such as
cron
jobs would be to save a copy of it on your system where
cat
normally lives, making sure that
the "execute" file mode bits are set
.



There are certainly other widespread tools, such as Perl, that could also be used in place of
cat
. However, I wanted to focus only on utilities that are so ubiquitous that they appear in the POSIX standard. It should be noted that even some of these, such as
compress
and
uuencode
, aren't always present on modern systems.



References:







Cat specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/cat.html





Definitions: Text File
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_403





Nl specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/nl.html





Read specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/read.html





Tr specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/tr.html





Tee specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/tee.html





Ed specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/ed.html





Ex specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/ex.html





Comm specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/comm.html





Paste specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/paste.html





Tail specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/tail.html





Sort specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sort.html





Dd specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/dd.html





Compress specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/compress.html





Uuencode specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/uuencode.html





Sed specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html





Grep specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/grep.html





Awk specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html





Cut specification
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/cut.html







Provide feedback on this episode.

This show has been flagged as Clean by the host.

Python and its libraries

1. librosa — loading and resampling audio

def load_audio(path):
y, sr = librosa.load(path, sr=SR, mono=True)
return y, sr

Every episode, regardless of its original sample rate or channel count, gets loaded down to a single mono stream at 16kHz (SR = 16000). That's one line of librosa, resampling, and folding stereo to mono. It made every downstream tool (VAD, pitch tracking, MFCC extraction) operate on a consistent, predictable input regardless of what an MP3 encoder or podcast host originally used.

2. parselmouth (Praat bindings) — pitch and voice-quality tracking

def pitch_and_hnr(y, sr):
snd = parselmouth.Sound(y, sampling_frequency=sr)
pitch = snd.to_pitch(time_step=PITCH_TIME_STEP, pitch_floor=PITCH_FLOOR, pitch_ceiling=PITCH_CEILING)
f0 = pitch.selected_array["frequency"]
times = pitch.xs()
try:
harmonicity = snd.to_harmonicity_cc(time_step=PITCH_TIME_STEP)
hnr = np.array([harmonicity.get_value(t) for t in times])
hnr = np.nan_to_num(hnr, nan=-999.0)
exceptException:
hnr = np.full_like(f0, 999.0) # fail open if HNR extraction errors
return f0, hnr, times

parselmouth is a Python wrapper around Praat, the standard tool in phonetics research. This is the heart of the whole vocal-monotony measurement: to_pitch() gives fundamental frequency (F0, roughly "pitch") every 20ms, and to_harmonicity_cc() gives harmonics-to-noise ratio (HNR) at the same timestamps — a signal-quality score used later to throw out pitch estimates taken from noisy or badly-compressed audio rather than let bad audio masquerade as "flat" pitch.

3. webrtcvad — deciding which frames are actually speech

def vad_speech_mask(y, sr, aggressiveness=2):
vad = webrtcvad.Vad(aggressiveness)
pcm16 = (np.clip(y, -1.0, 1.0) *32767).astype(np.int16)
n_frames =len(pcm16) // VAD_FRAME_LEN
mask = np.zeros(n_frames, dtype=bool)
for i inrange(n_frames):
chunk = pcm16[i * VAD_FRAME_LEN:(i +1) * VAD_FRAME_LEN]
raw = struct.pack("%dh"%len(chunk), *chunk)
try:
mask[i] = vad.is_speech(raw, sr)
exceptException:
mask[i] =False
return mask, times

webrtcvad is the voice-activity detector out of Google's WebRTC project — a frame-by-frame classifier ("is this 30ms chunk speech or not"). It needs raw 16-bit PCM bytes, not floats, hence the np.clip and struct.pack conversion.

4. librosa.feature — telling music apart from speech

flatness = librosa.feature.spectral_flatness(y=y, hop_length=hop)[0]
chroma = librosa.feature.chroma_stft(y=y, sr=sr, hop_length=hop)
onset_env = librosa.onset.onset_strength(y=y, sr=sr, hop_length=hop)
...
chroma_diff = np.mean(np.linalg.norm(np.diff(chroma_win, axis=1), axis=0))
chroma_stability =1.0/ (1.0+ chroma_diff)
...
scores[w] =0.4* chroma_stability +0.35* periodicity +0.25*min(flat_mean *3, 1.0)

Podcast intros/outros are often music, and music has a very different pitch profile than speech — including it would bias the "monotone" measurement. Rather than train a classifier, three librosa features were combined with hand-tuned weights: spectral flatness (music tends to have more tonal structure, less noise-like spectrum), chroma stability (sustained harmony vs.

constantly-changing speech formants), and onset periodicity (a regular beat vs. speech's irregular rhythm).

5. librosa.feature.mfcc + sklearn.cluster.KMeans + sklearn.metrics.silhouette_score — an unsupervised speaker-count attempt

mfcc = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=13, hop_length=hop)
...
chunk_vecs.append(np.concatenate([seg.mean(axis=1), seg.std(axis=1)]))
...
X = np.array(chunk_vecs)
X = (X - X.mean(axis=0)) / (X.std(axis=0) +1e-9)

best_k, best_score =1, -1.0
for k inrange(2, min(MAX_SPEAKERS, len(X) -1) +1):
labels = KMeans(n_clusters=k, n_init=10, random_state=0).fit_predict(X)
iflen(set(labels)) <2:
continue
score = silhouette_score(X, labels)
if score > best_score:
best_k, best_score = k, score

estimated = best_k if best_score >= SILHOUETTE_MIN else1

MFCCs (Mel-frequency cepstral coefficients) summarize the shape of a short audio window's spectrum — a rough "voice fingerprint" widely used in speech tasks. The idea: chop each episode into 2-second chunks, turn each into a fingerprint vector, standardize (z-score) the features, then use KMeans to see if the chunks naturally split into 2+ clusters (different voices) rather than 1 (a single voice throughout). silhouette_score scores how well-separated the clusters are, and picks the best k — or falls back to "1 speaker" if no clustering clears a minimum quality bar. This part of the investigation did not work, as it picked up noise as another host.

6. scipy.stats.mannwhitneyu + a hand-written Cliff's delta — comparing two groups without assuming a normal distribution

def cliffs_delta(a, b):
a, b = np.asarray(a), np.asarray(b)
gt =sum((x > y) for x in a for y in b)
lt =sum((x < y) for x in a for y in b)
return (gt - lt) / (len(a) *len(b))


def compare_group(df, metric, label):
hpr = df[df.corpus =="hpr"][metric].dropna()
cmp= df[df.corpus =="comparison"][metric].dropna()
u, p = stats.mannwhitneyu(hpr, cmp, alternative="two-sided")
delta = cliffs_delta(hpr.values, cmp.values) iflen(hpr) *len(cmp) <200000elseNone

scipy.stats.mannwhitneyu is a statistical test: it compares two groups of numbers (HPR vs. comparison podcasts, per metric) without assuming they're normally distributed — appropriate for skewed, real-world data like pitch-range measurements. scipy doesn't ship an effect-size measure alongside it, so Cliff's delta — "of all possible (HPR episode, comparison episode) pairs, how much more often is HPR's value the smaller one?" This pairing (a significance test plus an effect-size number) was used throughout because a low p-value alone doesn't say whether the difference is actually big enough to matter.

7. pandas — grouping, stratifying, and cross-tabulating results

print(df.groupby("corpus")["retained_fraction"].describe())
...
print(pd.crosstab(df["corpus"], df["register"]))

Once data/results.jsonl (one JSON object per analyzed episode) is loaded into a pandas.DataFrame, two one-liners answer two different questions that would otherwise need custom code: .groupby(...).describe() gives a distribution summary (count, mean, std, quartiles) of audio-quality coverage per corpus, in one call, and pd.crosstab() cross-tabulates HPR vs. comparison against male-typical/female-typical register — a quick sanity check that the two collections of podcasts had a similar gender-register mix.

8. matplotlib — box plots for the headline comparison

fig, ax = plt.subplots(figsize=(6, 4))
data = [df[(df.corpus == c) & df[metric].notna()][metric].values for c in ["hpr", "comparison"]]
ax.boxplot(data, tick_labels=["HPR", "Comparison"], showmeans=True)

A box plot per pitch-range metric, HPR next to comparison podcasts, mean marked as well as the median.

9. spacy word vectors — measuring topic drift, not just word choice

nlp = get_nlp() # spacy.load("en_core_web_md")
...
chunk_docs = [nlp(c) for c in chunks]
chunk_vecs = []
for d in chunk_docs:
v = d.vector
norm = np.linalg.norm(v)
if norm >1e-6:
chunk_vecs.append(v / norm)
...
adj_sims = [float(np.dot(V[i], V[i +1])) for i inrange(n -1)]

Each ~130-word chunk of a transcript is embedded as a single vector using spaCy's medium English model (en_core_web_md, which has built-in word vectors), then L2-normalized so a dot product between two vectors
is
their cosine similarity. Comparing consecutive chunks' similarity gives a "does the topic barely move between one chunk and the next" score.

10. lexicalrichness — vocabulary diversity that doesn't cheat on transcript length

lr = LexicalRichness(text)
try:
mtld = lr.mtld(threshold=0.72)
exceptException:
mtld =None

A naive vocabulary-diversity score (unique words ÷ total words) gets worse the longer a text runs, because of how the metric is defined — not because the speaker is actually repeating themselves more.

MTLD (Measure of Textual Lexical Diversity) avoids that by sweeping through the text and counting how many words it takes before diversity drops below a threshold, then averaging — a metric designed specifically to stay comparable between a 10-minute and a 40-minute episode.

11. vaderSentiment — a lexicon-based sentiment score

vader = get_vader() # SentimentIntensityAnalyzer()
sentiments = [vader.polarity_scores(c)["compound"] for c in chunks]
...
"sentiment_std": float(np.std(sentiments)) iflen(sentiments) >1elseNone,

VADER scores each chunk's sentiment from -1 to +1 using a hand-built lexicon and rule set (not a neural network) — chosen for being fast, deterministic, and cheap enough to run per-chunk across hundreds of episodes. The value then used isn't the average sentiment — it's the
standard deviation
of sentiment across an episode's chunks: does the emotional

tone ever move, or does it stay flat throughout?

12. zlib (standard library) — compression ratio as a gauge for entropy

def compression_ratio(text):
raw = text.encode("utf-8")
iflen(raw) <200:
returnNone
compressed = zlib.compress(raw, level=9)
returnlen(compressed) /len(raw)

No third-party NLP library involved at all — just the standard-library zlib module. A gzip-style compressor finds and exploits repeated patterns; if it can squash a transcript down a lot, that transcript was relatively repetitive and predictable, and if it can barely compress it, the wording was more varied. This is roughly tells you "verbal monotony."

13. re (standard library) — turning "does HPR favour solo hosts" into a checkable heuristic

MULTI_PRESENTER_PATTERNS = [
r"\bco-?host(?:s|ed|ing)?\b",
r"\bguest\b",
r"\binterview(?:s|ed|ing)?\b",
r"\btalks? with\b",
r"\bjoins? me\b",
...
]
CORRESPONDENT_LINK_RE = re.compile(r'correspondents/(\d+)\.html')

def multi_presenter_score(summary, notes):
text = (summary or"") +" "+ (notes or"")
hits = [p for p in MULTI_PRESENTER_PATTERNS if re.search(p, text, re.IGNORECASE)]
summary_links =set(CORRESPONDENT_LINK_RE.findall(summary or""))
return {
"flagged_multi_presenter": bool(hits) orlen(summary_links) >1,
}

Rather than the acoustic speaker-clustering attempt (item 5, which turned out to be unreliable), a more trustworthy stat came from plain text: HPR's own episode metadata (pulled from a local knowledge-base repo) has host-written summaries, and a short list of regex patterns ("guest", "interview", "joins me", etc.) plus counting linked correspondent profiles was enough to flag likely multi-presenter episodes.

14. multiprocessing.Pool — running the same analysis over hundreds of files without waiting hundreds of times over

work = [(r, local_paths[r["id"]]) for r in batch]
with mp.Pool(args.workers) as pool, open(args.out, "a") as out_f:
for result in pool.imap_unordered(_worker, work):
out_f.write(json.dumps(result) +"\n")
out_f.flush()

Every batch pipeline (acoustic pitch analysis, speaker-count pass) used the same shape: Pool(workers).imap_unordered(...) to run the per-episode function across multiple CPU cores at once, streaming results out to a JSON-lines file as each one finishes rather than waiting for the whole batch. imap_unordered was chosen over map so a single slow episode couldn't hold up writing out the results that had already finished.

15. subprocess + a resumable batch loop — working within a disk-space limit

def load_done_ids(results_path):
ifnot os.path.exists(results_path):
returnset()
done =set()
withopen(results_path) as f:
for line in f:
...
done.add(json.loads(line)["id"])
return done

...
rows = [r for r in rows if r["id"] notin done]
...
for batch_start inrange(0, len(rows), args.batch_size):
batch = rows[batch_start:batch_start + args.batch_size]
local_paths = fetch_batch(batch, args.scratch) # scp from the remote host
...
for p in local_paths.values():
os.remove(p) # delete before next batch

All the podcasts (~220GB) didn't fit on local disk (~54GB free). The fix was a loop: scp (via subprocess) one batch of audio in, analyze it, delete it, move to the next batch. load_done_ids() makes the whole pipeline resumable by reading back what's already in the output file, so a crash or a deliberate stop partway doesn't lose completed stuff.

Links

https://news.ycombinator.com/item?id=49110547

https://mastodon.social/@ngate/117009495478432200

Ah yes, the riveting world of Hacker Public Radio: a quaint little corner where tech wizards ✨️ divulge their magical secrets ✨️ to each other in the most riveting monotone known to man. Who knew endless "community" podcasts could be so... utterly riveting? 📻️🎙️ Apparently, everyone is just dying to contribute to this cacophony once a year—what a privilege! 😂

https://hackerpublicradio.org/

#HackerPublicRadio #techCommunity #podcasting #MonotoneMagic #rivetingContent #HackerNews #ngated


Provide feedback on this episode.

This show has been flagged as Clean by the host.












BruCON: https://brucon.org





BruCON
is an annual security and hacker conference providing two days
of an interesting atmosphere for open discussions of critical
infosec issues, privacy, information technology and its
cultural/technical implications on society. Organized in
Belgium, BruCON offers a high quality line up of speakers,
security challenges and interesting workshops. BruCON is a
conference by and for the security and hacker community.

​The conference tries to
create bridges between the various actors active in computer
security world, included but not limited to hackers, security
professionals, security communities, non-profit organizations,
CERTs, students, law enforcement agencies, etc…..




https://www.youtube.com/@brucontalks/videos




Provide feedback on this episode.

This show has been flagged as Clean by the host.

In this episode we look at a very old style of beer known as Mild. Paul, in the absence of any locally available Milds has a Guinness, Dave tastes
Banks's Mild
and Kevie Samples
1950s Burton Ruby Mild
.







Connect with the guys on
Untappd
:







Dave






Paul






Kevie




















The intro sounds for the show are used from:







https://freesound.org/people/mixtus/sounds/329806/






https://freesound.org/people/j1987/sounds/123003/






https://freesound.org/people/greatsoundstube/sounds/628437/










The next 3 beer styles to be reviewed:









Bitter



Pale Ale



Trappist







Provide feedback on this episode.