I’ve been struggling with a rather complex shell script, and it’s becoming apparent that Bash might not be the best choice for this particular task. While I usually gravitate towards statically typed languages like Go or Rust, I’ve noticed that many people recommend alternative languages such as Lua or Python for scripting tasks.
I’m curious to know your opinions and experiences with scripting languages for larger or more intricate shell scripts. Have you ever encountered a situation where Bash just didn’t cut it, and if so, which scripting languages did you turn to for a more effective solution? Are there any specific languages you found particularly suitable for debugging, testing, or handling complex logic in your shell scripts?
When I’m doing too much to maintain in bash, and not enough to merit Python, I use PowerShell.
My controversial take: if you’re looking for a better scripting language and haven’t tried PowerShell, you should give it a try.
It’s weird that Microsoft made a real shell.
PowerShell is actually open source, and it runs everywhere, including Mac and Linux. On Windows and Ubuntu, it’s already installed.
Powershell’s quality JSON and CSV handling is a huge game changer for quick scripts. The webrequest module is high quality. File operations are a breeze. Unlike bash, PowerShell can be formated to be pretty readable, when you care. Environment variable handling is mildly improved. Resusable code via modules is huge for quality of life.
PowerShell is the bash rewrite with lessons learned we all have wanted, but it’s not on a lot Linux folks radar because Microsoft published it.
Alternatively try nushell. Its basically powershell without ms
Oh, nice. I’ll check it out.
I’m trying to understand where you’d want to use PowerShell over Python. What’s something that’s in-between bash and Python?
Good question.
I choose PowerShell over Python when I need to call out to an existing command line utility, because I find the Python subprocess module is a huge pain in the ass.
PowerShell has about 80% of the power and readability of Python, while actually being a native shell.
Yeah, that’s fair. I was wondering if you’d call that out.
Popenis rather opaque. I don’t know that I’d go so far as to try to remember yet another language to avoid it. I respect the decision though, especially with the portability of modern PowerShell.
This 100%. I liked bash scripting when I was in college. Took some time to actually learn powershell, and it’s been amazing. Steep learning curve in the beginning, but it’s worth it
Edit: I had also meant to mention the fact that powershell definitely feels a little similar to bash. I mean, not quite the same. Powershell is more object oriented, whereas bash is more text oriented. Powershell for structured data is nice.
not to mention that you have all the dotnet ecosystem at your fingertips. Wanna write a WPF application? Go ahead. You want to use ML.net? A bit clunky but doable.
Couldn’t agree more. It’s a great shell and scripting language. It’s object-oriented nature, native support for virtually every text format (csv, json, xml) and great libraries for others (yaml, excel), awesome regex and web/rest services support… it’s hard to beat and works on virtually every platform.
Too few people in the Linux community will even look at it though since it has MS name on it.
For years my go to (after Bash) was Python. However, in the last few years I’ve switched to Rust for any kind of shell command wrapper or CLI tool.
TL;DR I think Rust is best suited to more complex CLI work.
Exact same story here. Bash -> Python -> Rust.
Generally speaking, people should settle on a compiled language if they can. They can iterate as fast as interpreted languages these days.
Edit: If you want to try something different in scripting, try the execline language. Its interpreter processes the script and exits immediately even before the script execution begins. Traditional shell interpreters (like bash) stay active till the entire script is finished. Execline achieves this by a clever chaining of Unix execs, forks and variable substitutions. This makes execline scripts lighter (useful in embedded systems), more secure and less error-prone than traditional scripts. The downside is that writing them will feel a bit weird - since the fundamental paradigm is different from regular shells. However, that will be a refreshing change if you’re someone who likes to experiment and try new things.
Yes, and Rust with incremental compilation is pretty fast to iterate as well, as long as you don’t use massive libraries/build-scripts etc.
Would you have any example (not necessarily yours) to showcase this? I mean, how is it better suited than say, C++?
Not op, but I feel the same as them.
Compared to C++, Rust has a very good toolchain and libraries. With C++ setting up a project that has dependencies is… painful. I’m a full-time C++ programmer with over 8 years of experience and if I didn’t have to, I would never choose it for something new.
With Rust creating a new project and adding dependencies is trivial. There are a lot of great libraries and the ease with which you can use them is very empowering.
Clap and serde are super powers for CLI programs 😀For smaller scripts that don’t yet “deserve” full rust treatment, I now use nushell for personal projects.
I too use Rust for what normal people use shell scripts for. But I have a feeling that Rust is falling into the same trap that other languages with similar easy dependency management fall into (Python and NPM are good examples). You end up with a dozen direct dependencies and hundreds of indirect ones with dozens of levels of hierarchy. C and C++ programs have fewer dependencies because each additional one adds more headache for the developer. Drew Devault’s Hare language is giving language repo and package manager a skip for the same reasons. And I’m starting to think that he may have a point.
I think it’s not that bad yet, when comparing with npm. Usually the dependencies I use are of very high quality. But I’m also very selective with dependencies. I’m rather writing a simple part myself, than using a not-really maintained low-quality dependency…
Btw. I have not looked into the Hare language yet (will do that now), but if it’s similar as deno, I won’t like it. You want to have some kind of package management IME…
Drew Devault’s Hare language
Ok, they say “use your distros package-manager”, that’s basically asking for the same disaster as C or C++. I think cargo is one of the selling points of Rust.
At least say something like we use “Nix” for default package-management (which does a lot of things right)…
I personally don’t have any real experience with Go. Lots of smart folks I work with love it. In general, most of what I have read suggests that Rust is better suited to CLI tooling. For my use case it came down to:
- Rust’s cargo system
- The clap crate (which supports building out bash shell completion scrips via a Rust build script). Basically means I can generate a completion script at compile time and include this in the package I distribute to users)
- Rust’s out of the box performance
- The heavy lifting done by the borrow checker in bringing safety
Just curious have you tried Go for this? Go was recently approved at work and I have seen articles about Go for things like this and just wondering if it is worth it. I have been using ansible and chef but need to explore other options. I want to use Rust but I know the road blocks I will have to work through at work. So just wondering if you had any insights to Go over Rust
Not him, but I much more like the type-system of rust (e.g. enums).
It’s especially true when you want to parse some json/xml/whatever. Just describe your datastuctures with regular struct and enum, add serde and done! It’s like magic!
If we remove words “serde” and “enum”, no one will be able to guess whether the argument is for rust or golang.
I personally don’t have any real experience with Go. Lots of smart folks I work with love it. In general, most of what I have read suggests that Rust is better suited to CLI tooling. For my use case it came down to:
- Rust’s cargo system
- The clap crate (which supports building out bash shell completion scrips via a Rust build script. Basically means I can generate a completion script at compile time and include this in the package I distribute to users)
- Rust’s out of the box performance
- The heavy lifting done by the borrow checker in bringing safety
for larger or more intricate shell scripts
Those are call applications. Use any language you like. If go/rust is what you know use them. I use rust all the time for things beyond run a bunch of commands and tends to be my go to when I need to process data in any way.
If the script gets too large or you think it will, write a full program.
With languages like python or go its not that hard either and you get alot of benifits aswell.Other than that, use whatever languwge works and you know.
Unpopular opinion: I’m old school and would probably use perl.
I do most of my scripting in perl too. Python has always irked me.
Only semi-related: I recently switched from bash shell to fish. I should have don’t that years ago. Don’t know why I held on to bash so tight.
What do you like about fish? Have you tried zsh? I’m in the market for a new shell too.
I’m not the person to say anything about zsh vs. fish. I last tried zsh around 2008. Back then I decided to stick with bash over other shells. At the time (and for decades earlier) it was clear that sh was inadequate. So the Bourne Again Shell was (and still is) ubiquitous. Other shells fighting for user space seemed like the xkcd-927 problem.
Now, I’m basically seeing that if I’m stuck without my .bashrc file, installing fish gives me most of the niceties I like. It also gives me niceties that I wouldn’t have been able to do, like nice multi-lining. And while I personally find fi and esac charming, they are pretty dumb. And I certainly don’t miss the for-do-done construction.
I bet I would like zsh as much as fish, but I don’t have any motivation to try it.
fish looks at my path to highlight mistyped commands. It autocompletes on the fly and autocompletes with attention paid to the usage in your history. The coolest thing is that it parses man pages. That allows autocomplete to know the options of a command as long as it has a proper man page (which it just should).
This is the correct answer. Perl is shell-like with support for advanced data structures and data parsing capabilities. Modern Perl is very slick, especially with the new object system.
Modern Perl
Perl or Raku?
Both are good and they each have their uses.
Perl is very Unix-y, recent releases have a very good object system, and Perl is quite fast but the syntax can take some getting used to. CPAN is a huge database of Perl modules, you’ll likely find what you need module wise.
Raku is amazingly flexible and I like its object and type systems more than other languages. The only only down side is compared Perl is that Raku on the slow side, even Python is faster at the moment. Raku has a much more consistent syntax than Perl but the module ecosystem is nowhere near as big.
I’d say try both and use what seems to be the most optomal for whatever task you’re dealing with. Personally, I use both for quick scripts about equally with performance and module availability usually being the deciding factors.
–
- https://wiki.gentoo.org/wiki/Perl (Scroll down to the External Links section for learning resources.)
- https://raku.guide/
My personal favourite remains perl for anything text oriented or for simple-ish orchestration.
I love Raku (formerly perl 6) for this purpose as well.
I was in a similar situation not too long ago.
My criteria for another scripting language included that it should be preinstalled on all target systems (i. e. Debian and Fedora), it should be an interpreted language and it needs to have type safety.
Afterall I settled with Python due to its popularity, its syntax and features (type safety since v3.6, etc.) and the fact that it is preinstalled on many Linux distributions. System components often use Python as well, which means that libraries to interact with the system tend to be included by default.
pyinvoke.
You can create quick and dirty CLIs, invoke shell commands, and have all of python available for things like parsing config files, getting and setting environment variables, and making remote REST calls.
Totally second this.
But I 'd like to recommend starting with subprocess module which is built-in. Then go ahead and see what the extensions have to offer… There are other that may come handy, like Typer from Tiangolo, and Fire from Google.
Personally, I don’t feel like it’s worth learning a separate scripting language when you’re comfortable with a full-fledged programming language.
Python, Lua etc. used to be on a whole different level of usability, when compared to C. But compared to modern, high-level languages, the difference is marginal. Not to mention that at least compared to Rust, they start to look rather antique, too, and lack in robust tooling.
If you don’t feel like maintaining a whole git repo, use e.g. rust-script.
A shell script can be more concise if you’re doing a lot of shell things. Keeps you from having
os.system()all over the place.Things like “diff the output of two programs” are just more complex in other languages.
I love rust, but replacing my shell scripts with rust is not something I would consider doing any more than I’d consider replacing rust with my shell scripts.
Oh, I didn’t mean to say, you should throw out your shell scripts. For anything less than, say, 20 lines, they’re perfectly appropriate.
I’m saying, Rust et al start to feel like a good choice from, say, 100 lines upwards, and I just don’t think, it’s worth bridging the gap between those two.
In particular, you can build a function that allows you to run commands without much boilerplate, e.g.:
run("echo hello | tee out.txt");
(The implementation just appends that argument toCommand::new("sh").arg("-c")and runs it.)That way, you can do the more complex things in Rust, whether that’s control flow or something like modifying a JSON file, without giving up the utility of all the CLI tools on your system…
Somewhat of a weird addendum, but I actually only realized, you could port directly over like that, while writing the above comment.
Now I actually tried it on a 22 lines long shell script that I’ve been struggling with, and holy crap, I love it.
Like, I should say that I have always been (and likely will always be) shit at shell scripting. Any time I wanted to do a basic
if, I had to look up how that works.
As a result, even those 22 lines were ripe with code duplication and I always felt really unsure about what will actually happen during execution.Well, rightfully so. While porting over, I realized I had a bug in there, which has been annoying me for a while, but I always thought, well, it is a shitty shell script. I still remember thinking, I should probably not implement it like that, but then leaving it anyways, because I felt it would become unreadable with the shell syntax.
Now it actually feels maintainable, like I can even easily expand on it.
And I have to say thatrust-scriptis really smooth. I barely notice that it’s compiling during the first run after changing the script file, and it’s fully cached afterwards, so it executes instantly.I’ll still have to check for libraries that basically provide such a
run()function/macro for me, but yeah, basically my threshold for not using shell scripts just dropped to any kind of control flow being involved.Yeah the strict type-system of Rust is great at finding issues.
I think when understanding, that bash is basically only programs with parameters (
[is a program that takes all kinds of parameters and as last parameter]) then bash is quite ok for stuff that doesn’t need a lot of algorithms, i.e. passing the in and out from one program to another. But as soon as there’s basic logic, You’ll want to use a fully-fledged programming language.Also the maintainability aspect: You can just start using fancy stuff you never want to use in bash and it can slowly grow into a library or application or something like that.
Btw. I have started a syntax-sugar library/crate that creates typing information for all kinds of programs via the builder-type-state-pattern, so that you don’t always have to look up
manetc. and that it should be more convenient to execute programs (not open sourced yet, and low priority for me as I’m working on various other exciting projects currently)
Yeah as weird as it sounds to use a “low”-level systems programming language such as Rust. Rust works surprisingly well as “script” language. (And you don’t have to deal with the ugliness of bash, admittedly though, that bash is quite a bit more concise when using a lot of program executions and piping the results etc.)
I make bash scripts to automate the configuration of new servers. Stuff like install packages, create users, create groups, configure the database, manage permissions…
I feel like that sort of stuff would be a nightmare to do in high level languages but maybe I’m just too used to bash.
As I already responded to others, my comment was meant in the context of the question, so I would not learn a scripting language in addition to Bash + a programming language.
For just running commands one-after-another, Bash is basically a minimal encoding, so no reason not to use it.
When you do start to need if-elses, loops etc., that’s where Bash starts to become somewhat difficult to read. And personally, as someone who’s not fluent in Bash control flow, I found it quite useful to do the control flow in my programming language of choice, but still just calling commands like you’d do in Bash.
Of course, this is a non-standard setup, and most target hosts will have Bash pre-installed, not rust-script, so it does obviously make a lot of sense to continue using Bash for what you’re doing.
In general, my comment was meant for programmers. An ops person might know a full-fledged programming language, but still want to learn Python, because they need to write tons of Ansible tasks or whatever.
What are you trying to achieve in Bash that you’re struggling with? It’s hard to suggest alternatives without knowing your objectives since languages excel in different areas.
Temporary files management for a
~/tmpfolder, with archiving and cleanup after x and y days.You could look at logrotate if you don’t want to do something custom
The expression syntax for the GNU find command is very powerful. I would expect that it is up to the task. If you don’t have the GNU find command with it’s extensions I could see how it’s would be difficult.
As @damium@programming.dev says you may be able to do this with
findcommand. This command lists all PDF files under ~/tmp that were created more than 7 days ago and does a directory listing. You could use this as a basis to move create an archive of individual files.find ~/tmp -ctime +7 -iname "*pdf" -exec ls -rlht {} \;The
findcommand also has a-deleteflag.I have in the past used this combination to implement file management. I don’t have access to the script any more. I don’t remember why we used a shell script rather than logrotate as per @oddityoverseer@lemmy.world
deleted by creator
I usually use Awk to do the heavy lifting within my Bash scripts (e.g. arg parsing, filtering, stream transformation), or I’ll embed a Node.JS script for anything more advanced. In some cases, I’ll use eval to process generated bash syntax, or I’ll pipe into sh (which can be a good way to set up multiprocessing). I’ve also wanted to try zx, but I generally just stick to inlining since it saves a dependency.
Look at Raku. This seems to be where Raku shines relatively brightly.
I think vanilla perl is also worth looking at if you need the script to be portable
Try NuShell
I’ve used C# in the past. There’s a tool called dotnet-script that lets you run csharp files as scripts without having to compile.
It does require you to install dotnet though.
But tbh it’s pretty easy to just chuck an executable together.
