Fixing the LS_COLORS problem and improving terminal colors

[loman]

The LS_COLORS environment variable determines file colors during directory listings (ls colors, get it?). LS_COLORS is used to set a file's listing color based on a pattern match to the file name. The default file colors are set by the utility dircolors, and these colors are boring. Many people agree.

You can find some much nicer coloring schemes on the net. But despite this attention to detail, there isn't an easy way to change coloring schemes, or to even set them for that matter. Most people don't know about these other color schemes and just use the default.

To make this feature more accessible, I created the utility set_file_colors to graphically demo and change LS_COLORS among various fancy options. If you use this utility, you will be all set for beautiful and informative listings.


HOWEVER, stuffing these fancy coloring schemes into LS_COLORS frequently makes the environment variable too large. Environment variables weren't designed with this in mind. Having a large environment variable causes some programs to bomb with the message "ERROR: too long environment variables please use --rmenv", or words to that effect. Even worse, some Linux programs launched from a menuing system will silently fail to run. This is because stderr is redirected, and you can't see the error message.

I was trapped between having beautiful file listings and needing all programs to run without bombing.

I solved this problem by rewriting the GNU coreutils ls command and the tree program (the two main programs using LS_COLORS). My modified programs can now store a file name in LS_COLORS that contains the color scheme. Under this new system, which is a trivial modification of the old system, three new hidden files are added to each user's $HOME directory: .dircolors, .dircolors_selection, and .dircolors_setting. The .dircolors is a script that sets LS_COLORS based on the settings in .dircolors_setting. The file .dircolors_selection actually contains the LS_COLORS, as in echo $LS_COLORS > ~/.dircolors_setting. Thus, the color naming scheme remains unchanged. This makes the LS_COLORS variable very short, and programs no longer bomb because of the LS_COLORS being too long.

The magic is that the LS_COLORS variable may now express a path to a file. If LS_COLORS begins with a ';', which is malformed under the previous rules, the variable is interpreted as a file path. In this way, should the user choose a more traditional LS_COLORS setting, everything will work as before. Under the new system, LS_COLORS=';/home/user/.dircolors_setting' is a valid form.

You can find links and instructions for the updated programs in the build notes below.

While these new color schemes are nice, they don't cover a common situation. Sometimes, the files in a directory are of similar types, which are colored similarly. So you see a giant wall of similar colors. But what you really want is to see a big difference between file types. To address this issue, I added ls-rainbow and ls-normal programs. ls-rainbow compiles a histogram of the file types (by extension). It then gives those files pre-assigned vibrant colors that differentiate them. As a result, files which might appear the same color under a typical color scheme now have radically different colors. ls-rainbow resets LS_COLORS to accomplish this. ls-normal switches the LS_COLORS variable back to the previous setting. ls-normal is just an alias to

Code: Select all

export LS_COLORS=`dircolors` && [[ -f $HOME/.dircolors ]] && source $HOME/.dircolors; ls

Examples of ls-rainbow vs ls-normal:



The new programs also respect a new environment variable: TERM_BACKGROUND. You can set TERM_BACKGROUND to light or dark, and the color scheme will adjust automatically. This allows a single color setting to work when you change terminal background colors. UNFORTUNATELY, none of the terminal emulators know about this feature, so you will need to set it manually after changing your terminal background. In the future, I will be adding this feature to xfce4-terminal.

This new system is a temporary fix to solve the immediate problems. I can't stress that enough. I have a much more detailed scheme to address file coloring issues. However, this will need to wait until I can rewrite the coreutils in C++. The current coreutils is spaghetti code, and belongs to a different time. It needs a major re-think that doesn't break backward compatability. I only fixed the C code because I didn't realize how bad the current code was. In the amount of time I spent, I could have been well on the way to writing in C++. I see that there is a project to rewrite coreutils in the latest messiah-language (Rust). For various reasons, this isn't going to fly. I'm sure it will work, but it will be equally unmaintainable as the current situation, and no real improvement on the basic problems of the 21st Century that haven't been addressed by the GNUs.


Notes on building ls and tree:
The general idea is this: you will have the gnu coreutils and the tree package already installed. You download (or clone) those packages, substitute the new code, build, then clobber the existing ls and tree by copying ls and tree executables to the bin directory. You can either tell your package manager to lock the versions of these packages or you have to remember to rebuild and re-clobber as new versions of these packages come out.

The new ls can be built by overwriting ls.c with this new ls.c in any recent GNU coreutils and running make. So, it's a drop-in replacement. Similarly, Download tree, and overwrite color.c. Just build as normal. After chmod'ing the new ls and tree, copy the new ls to /bin/ls and /usr/bin/ls on a Linux system. Prerequisites for this new system are: bash and dircolors, both of which are typically installed. Copy the tree executable to wherever it normally lives. I am currently clobbering an installed package for coreutils and tree without difficulty.

While I was under the hood with ls, I thought I would add in a "directories only" mode to ls. ls -/ and ls -/R now list only directories, no files. I had previously been using ls -lF | grep / to accomplish this, but this is so much better. Directories without subdirectories don't get listed with ls -/R.


Notes on installing set_file_colors and its ancillary programs:

You need to modify your $HOME/.bashrc. Search for the line containing "dircolors -b ~/.dircolors". Change the .bashrc file to read as follows:

Code: Select all

test -r ~/.dircolors || eval "$(dircolors -b)"
test -r ~/.dircolors && source ~/.dircolors

Also change the .bashrc file for all accounts that want to use this new system and the /etc/skel/.bashrc file (so that new accounts also have the new system available). The previous .bashrc presumes that someone might set up a .dircolors file in their home directory to specify colors. But since that never happens, might as well not pretend that it does.

You need to modify /etc/bash.bashrc by adding the following line to the end of the file

Code: Select all

trap "{ source "{HOME}/.dircolors" 2> /dev/null; }" SIGUSR1

This trap of signal SIGUSR1 allows the set_file_colors to inform open bash shells of the color changes. It basically performs the equivalent of a SIGHUP for daemons. It's a shame that Linux doesn't have a standard way to force a program to re-read its configuration files, but it doesn't, and, my program needs to use SIGUSR1 to force every shell to re-read its .bashrc. If you don't add this line, the attempt by set_file_colors to update the running bash files will, instead, kill them all. Also, it may kill all your open shells the first time you use it, because .dircolors doesn't exist the first time. Make sure you open fresh terminals or source your .bashrc to refresh your settings.

Copy set_file_colors and ls-colors, ls-rainbow to /etc, and make them executable. Launching a yad program (set_file_colors) from the menu requires that the script be executable. Copy demo_colors.sh to /usr/local/bin and make it executable. Set_file_colors also requires an icon /usr/share/icons/gnome/48x48/categories/package_graphics.png, which is provided by the gnome icons package, which must be installed.

The main programs are called via shell aliases due to limitations of bash. You need to include the following lines (preferably) in your /etc/bash.bashrc (so everyone can use), or in your account ~/.bashrc or ~/.bash_aliases (assuming it is called by your bashrc somewhere):

Code: Select all

alias ls-rainbow='source /etc/ls-rainbow '
alias ls-normal='export LS_COLORS=`dircolors` && [[ -f $HOME/.dircolors ]] && source $HOME/.dircolors; ls'
alias set_file_colors='source /etc/set_file_colors';

If you want to access set_file_colors through a common desktop menuing system, you must copy set_file_colors.desktop to /usr/share/applications or your equivalent account directory so that the menuing system can find it. You may need to adjust a few things, but this should get you started.

Set_file_colors also calls tmux to demo the color schemes by opening windows with light and dark backgrounds. This was the only solution I could come up with to demonstrate the file colors before selecting. Hopefully, I can modify xfce4-terminal to serve this purpose and eliminate tmux in the future.

Because of the alias, ls-rainbow will be available to all users. It creates a histogram of files and tries to assign the greatest color differences among a limited list of colors that are perceptibly different. ls-rainbow has no help or man file, so I should mention that ls-rainbow also takes a second numerical argument, e.g. ls-rainbow N will construct the histogram on the top N directory levels. This is very useful for setting colors for the tree command.

ls-normal requires the new system, because it calls ~/.dircolors, which doesn't exist under the current system. Include in your .bash_aliases

Code: Select all

 alias ls-normal='export LS_COLORS=`dircolors` && [[ -f $HOME/.dircolors ]] && source $HOME/.dircolors; ls'

At some point, I will package all this stuff for Debian. But this is just the announcement that it exists. Waiting for feedback.

[jake-boards]

Uploaded a new ls.c

[jake-boards]

I've updated the code yet again, as well as the installation instructions. For anyone who tried to get this to work previously, I apologize for wasting your time.

I have tested the new instructions on MX19 and MX21 linux distros. Everything seems to work fine with the current instructions.

I realize that the install is difficult, but I haven't had time to package things. It's going to be challenging to say the least.

I want to remind everyone that if you only run this setup in one shell, your changes will not propagate to other shells until you open a new terminal UNLESS you inform all open bash shells (an option). This is by design. You may want different shells to have different colors. Also, if you read the .dircolors script, you will see that you can temporarily override the colors setting and use it in your own shell to set different colors in different shells.

Enjoy!