this post was submitted on 15 Jan 2025
73 points (98.7% liked)

Programming

17821 readers
531 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev



founded 2 years ago
MODERATORS
 

This may make some people pull their hair out, but I’d love to hear some arguments. I’ve had the impression that people really don’t like bash, not from here, but just from people I’ve worked with.

There was a task at work where we wanted something that’ll run on a regular basis, and doesn’t do anything complex aside from reading from the database and sending the output to some web API. Pretty common these days.

I can’t think of a simpler scripting language to use than bash. Here are my reasons:

  • Reading from the environment is easy, and so is falling back to some value; just do ${VAR:-fallback}; no need to write another if-statement to check for nullity. Wanna check if a variable’s set to something expected? if [[ <test goes here> ]]; then <handle>; fi
  • Reading from arguments is also straightforward; instead of a import os; os.args[1] in Python, you just do $1.
  • Sending a file via HTTP as part of an application/x-www-form-urlencoded request is super easy with curl. In most programming languages, you’d have to manually open the file, read them into bytes, before putting it into your request for the http library that you need to import. curl already does all that.
  • Need to read from a curl response and it’s JSON? Reach for jq.
  • Instead of having to set up a connection object/instance to your database, give sqlite, psql, duckdb or whichever cli db client a connection string with your query and be on your way.
  • Shipping is… fairly easy? Especially if docker is common in your infrastructure. Pull Ubuntu or debian or alpine, install your dependencies through the package manager, and you’re good to go. If you stay within Linux and don’t have to deal with differences in bash and core utilities between different OSes (looking at you macOS), and assuming you tried to not to do anything too crazy and bring in necessary dependencies in the form of calling them, it should be fairly portable.

Sure, there can be security vulnerability concerns, but you’d still have to deal with the same problems with your Pythons your Rubies etc.

For most bash gotchas, shellcheck does a great job at warning you about them, and telling how to address those gotchas.

There are probably a bunch of other considerations but I can’t think of them off the top of my head, but I’ve addressed a bunch before.

So what’s the dealeo? What am I missing that may not actually be addressable?

top 50 comments
sorted by: hot top controversial new old
[–] melezhik@programming.dev 5 points 2 hours ago* (last edited 1 hour ago) (1 children)

We are not taking about use of Bash in dev vs use Bash in production. This is imho incorrect question that skirts around the real problem in software development. We talk about use of Bash for simple enough tasks where code is rarely changed ( if not written once and thrown away ) and where every primitive language or DSL is ok, where when it comes to building of medium or complex size software systems where decomposition, complex data structures support, unit tests, error handling, concurrency, etc is a big of a deal - Bash really sucks because it does not allow one to deal with scaling challenges, by scaling I mean where you need rapidly change huge code base according changes of requirements and still maintain good quality of entire code. Bash is just not designed for that.

[–] Badland9085@lemm.ee 1 points 32 minutes ago

But not everything needs to scale, at least, if you don’t buy into the doctrine that everything has to be designed and written to live forever. If robust, scalable solutions is the nature of your work and there’s nothing else that can exist, then yeah, Bash likely have no place in that world. If you need any kind of handling more complicated than just getting an error and doing something else, then Bash is not it.

Just because Bash isn’t designed for something you want to do, doesn’t mean it sucks. It’s just not the right tool. Just because you don’t practice law, doesn’t mean you suck; you just don’t do law. You can say that you suck at law though.

[–] morbidcactus@lemmy.ca 1 points 6 hours ago (1 children)

I'm fine with bash for ci/cd activities, for what you're talking about I'd maybe use bash to control/schedule running of a script in something like python to query and push to an api but I do totally get using the tools you have available.

I use bash a lot for automation but PowerShell is really nice for tasks like this and has been available in linux for a while. Seen it deployed into production for more or less this task, grabbing data from a sql server table and passing to SharePoint. It's more powerful than a shell language probably needs to be, but it's legitimately one of the nicer products MS has done.

End of the day, use the right tool for the job at hand and be aware of risks. You can totally make web requests from sql server using ole automation procedures, set up a trigger to fire on update and send data to an api from a stored proc, if I recall there's a reason they're disabled by default (it's been a very long time) but you can do it.

[–] Badland9085@lemm.ee 2 points 5 hours ago (1 children)

People have really been singing praises of Powershell huh. I should give that a try some time.

But yeah, we wield tools that each come with their own risks and caveats, and none of them are perfect for everything, but some are easier (including writing it and addressing fallovers for it) to use in certain situations than others.

It’s just hard to tell if people’s fear/disdain/disgust/insert-negative-reaction towards bash is rational or more… tribal, and why I decided to ask. It’s hard to shake away the feeling of “this shouldn’t just be me, right?”

[–] morbidcactus@lemmy.ca 2 points 4 hours ago

I have to wonder if some of it is comfort or familiarity, I had a negative reaction to python the first time I ever tried it for example, hated the indent syntax for whatever reason.

[–] furrowsofar@beehaw.org 3 points 8 hours ago (1 children)

Just make certain the robustness issues of bash do not have security implications. Variable, shell, and path evalutions can have security issues depending on the situation.

[–] Badland9085@lemm.ee 2 points 6 hours ago (1 children)

Certainly so. The same applies to any languages we choose, no?

[–] furrowsofar@beehaw.org 2 points 5 hours ago* (last edited 5 hours ago)

Bash is especially suseptable. Bash was intended to be used only in a secure environment including all the inputs and data that is processed and including all the proccess on the system containing the bash process in question for that matter. Bash and the shell have a large attack surface. This is not true for most other languages. It is also why SUID programs for example should never call the shell. Too many escape options.

[–] zygo_histo_morpheus@programming.dev 12 points 14 hours ago (2 children)

One thing that I don't think anyone else has mentioned is data structures. Bash does have arrays and hashmaps at least but I've found that working with them is significantly more awkward than in e.g. python. This is one of several reasons for why bash doesn't scale up well, but sure for small enough scripts it can be fine (if you don't care about windows)

[–] Badland9085@lemm.ee 2 points 11 hours ago

That’s definitely worth mentioning indeed. Bash variables, aside from arrays and hashmaps that you get with declare, are just strings. Any time you need to start capturing a group of data and do stuff with them, it’s a sign to move on. But there are many many times where that’s unnecessary.

[–] syklemil@discuss.tchncs.de 3 points 13 hours ago

I think I mentioned it, but inverse: The only data type I'm comfortable with in bash are simple string scalars; plus some simple integer handling I suppose. Once I have to think about stuff like "${foo[@]}" and the like I feel like I should've switched languages already.

Plus I rarely actually want arrays, it's way more likely I want something in the shape of

@dataclass(frozen=True)
class Foo:
    # …

foos: set[Foo] = …
[–] vext01@lemmy.sdf.org 17 points 16 hours ago

Honestly, if a script grows to more than a few tens of lines I'm off to a different scripting language because I've written enough shell script to know that it's hard to get right.

Shellcheck is great, but what's greater is a language that doesn't have as many gotchas from the get go.

[–] ShawiniganHandshake@sh.itjust.works 14 points 20 hours ago (1 children)

I've worked in bash. I've written tools in bash that ended up having a significant lifetime.

Personally, you lost me at

reading from the database

Database drivers exist for a reason. Shelling out to a database cli interface is full of potential pitfalls that don't exist in any language with a programmatic interface to the database. Dealing with query parameterization in bash sounds un-fun and that's table stakes, security-wise.

Same with making web API calls. Error handling in particular is going to require a lot of boilerplate code that you would get mostly for free in languages like Python or Ruby or Go, especially if there's an existing library that wraps the API you want to use in native language constructs.

[–] Badland9085@lemm.ee 2 points 7 hours ago

This is almost a strawman argument.

You don’t have to shell out to a db cli. Most of them will gladly take some SQL and spit out some output. Now that output might be in some tabular format with some pretty borders around them that you have to deal with, if you are about the output within your script, but that’s your choice and so deal with it if it’s within your comfort zone to do so. Now if you don’t care about the output and just want it in some file, that’s pretty straightforward, and it’s not too different from just some cli that spits something out and you’ve redirected that output to a file.

I’ve mentioned in another comment where if you need to accept input and use that for your queries, psql is absolutely not the tool to use. If you can’t do it properly in bash and tools, just don’t. That’s fine.

With web API calls, same story really; you may not be all that concerned about the response. Calling a webhook? They’re designed to be a fire and forget, where we’re fine with losing failed connections. Some APIs don’t really follow strict rules with REST, and will gladly include an “ok” as a value in their response to tell you if a request was successful. If knowing that is important to the needs of the program, then, well, there you have it. Otherwise, there are still ways you can get the HTTP code and handle appropriately. If you need to do anything complex with the contents of the response, then you should probably look elsewhere.

My entire post is not to say that “you can do everything in bash and you should”. My point is that there are many cases where bash seems like a good sufficient tool to get that simple job done, and it can do it more easily with less boilerplate than, say, Python or Ruby.

[–] MonkderVierte@lemmy.ml 2 points 13 hours ago* (last edited 13 hours ago) (2 children)

Run checkbashisms over your $PATH (grep for #!/bin/sh). That's the problem with Bash.
#!/bin/sh is for POSIX compliant shell scripts only, use #!/bin/bash if you use bash syntax.

Btw, i quite like yash.

[–] Badland9085@lemm.ee 2 points 6 hours ago

Always welcome a new shell. I’ve not heard of yash but I’ll check it out.

[–] FooBarrington@lemmy.world 1 points 13 hours ago (1 children)

Any reason to use #!/bin/sh over #!/usr/bin/env sh?

[–] MonkderVierte@lemmy.ml 1 points 13 hours ago* (last edited 13 hours ago) (1 children)

I personally don't see the point in using the absolute path to a tool to look up the relative path of your shell, because shell is always /bin/sh but the env binary might not even exist.

Maybe use it with bash, some BSD's or whatever might have it in /usr without having /bin symlinked to /usr/bin.

[–] Badland9085@lemm.ee 1 points 6 hours ago (1 children)

There are times when doing so does make sense, eg if you need the script to be portable. Of course, it’s the least of your worries in that scenario. Not all systems have bash being accessible at /bin like you said, and some would much prefer that you use the first bash that appears in their PATH, e.g. in nix.

But yeah, it’s generally pretty safe to assume /bin/sh will give you a shell. But there are, apparently, distributions that symlink that to bash, and I’ve even heard of it being symlinked to dash.

[–] MonkderVierte@lemmy.ml 1 points 5 hours ago* (last edited 5 hours ago)

Not all systems have bash being accessible at /bin like you say

Yeah, but my point is, neither match they /usr/bin/env. Bash, ok; but POSIX shell and Python, just leave it away.

and I’ve even heard of it being symlinked to dash.

I think Debian and Ubuntu do that (or one of them). And me too on Artix, there's dash-as-bin-sh in AUR, a pacman hook that symlinks. Nothing important breaks by doing so.

[–] MITM0@lemmy.world 3 points 16 hours ago (1 children)

Well then you guys will love what this guy (by tha name "icitry") did with bash https://www.youtube.com/watch?v=b_WGoPaNPMY

He created a youtube clone with Bash

[–] Badland9085@lemm.ee 3 points 9 hours ago* (last edited 6 hours ago) (1 children)

That is definitely not something I would do… for work (totally not implying that I miiiight do it outside of work for shits and giggles :P).

I didn’t create this post trying to be like “y’all should just use Bash”, nor is it an attempt to say that I like Bash, but I guess that’s how people boil others down to these days. Fanatics only. Normalcy is dead. (I’m exaggerating ofc)

[–] MITM0@lemmy.world 2 points 7 hours ago

Basically, If you are crazy enough, you csn make anything with any language Hence, me sharing the video

[–] synae@lemmy.sdf.org 4 points 17 hours ago* (last edited 6 hours ago) (1 children)

As I've matured in my career, I write more and more bash. It is absolutely appropriate for production in the right scenarios. Just make sure the people who might have to maintain it in the future won't come knocking down your door with torches and pitchforks...

[–] Badland9085@lemm.ee 2 points 8 hours ago

That’s my take on the use of bash too. If it’s something that people think it’s worth bring their pitchforks out for, then it’s something you should probably not write in bash.

[–] FizzyOrange@programming.dev 18 points 1 day ago (2 children)

I'm afraid your colleagues are completely right and you are wrong, but it sounds like you genuinely are curious so I'll try to answer.

I think the fundamental thing you're forgetting is robustness. Yes Bash is convenient for making something that works once, in the same way that duct tape is convenient for fixes that work for a bit. But for production use you want something reliable and robust that is going to work all the time.

I suspect you just haven't used Bash enough to hit some of the many many footguns. Or maybe when you did hit them you thought "oops I made a mistake", rather than "this is dumb; I wouldn't have had this issue in a proper programming language".

The main footguns are:

  1. Quoting. Trust me you've got this wrong even with shellcheck. I have too. That's not a criticism. It's basically impossible to get quoting completely right in any vaguely complex Bash script.
  2. Error handling. Sure you can set -e, but then that breaks pipelines and conditionals, and you end up with really monstrous pipelines full of pipefail noise. It's also extremely easy to forget set -e.
  3. General robustness. Bash silently does the wrong thing a lot.

instead of a import os; os.args[1] in Python, you just do $1

No. If it's missing $1 will silently become an empty string. os.args[1] will throw an error. Much more robust.

Sure, there can be security vulnerability concerns, but you’d still have to deal with the same problems with your Pythons your Rubies etc.

Absolutely not. Python is strongly typed, and even statically typed if you want. Light years ahead of Bash's mess. Quoting is pretty easy to get right in Python.

I actually started keeping a list of bugs at work that were caused directly by people using Bash. I'll dig it out tomorrow and give you some real world examples.

[–] Badland9085@lemm.ee 1 points 7 hours ago (1 children)

I honestly don’t care about being right or wrong. Our trade focuses on what works and what doesn’t and what can make things work reliably as we maintain them, if we even need to maintain them. I’m not proposing for bash to replace our web servers. And I certainly am not proposing that we can abandon robustness. What I am suggesting that we think about here, is that when you do not really need that robustness, for something that may perhaps live in your production system outside of user paths, perhaps something that you, your team, and the stakeholders of the particular project understand that the solution is temporary in nature, why would Bash not be sufficient?

I suspect you just haven’t used Bash enough to hit some of the many many footguns.

Wrong assumption. I’ve been writing Bash for 5-6 years now.

Maybe it’s the way I’ve been structuring my code, or the problems I’ve been solving with it, in the last few years after using shellcheck and bash-language-server that I’ve not ran into issues where I get fucked over by quotes.

But I can assure you that I know when to dip and just use a “proper programming language” while thinking that Bash wouldn’t cut it. You seem to have an image of me just being a “bash glorifier”, and I’m not sure if it’ll convince you (and I would encourage you to read my other replies if you aren’t), but I certainly don’t think bash should be used for everything.

No. If it's missing $1 will silently become an empty string. os.args[1] will throw an error. Much more robust.

You’ll probably hate this, but you can use set -u to catch unassigned variables. You should also use fallbacks wherever sensible.

Absolutely not. Python is strongly typed, and even statically typed if you want. Light years ahead of Bash's mess. Quoting is pretty easy to get right in Python.

Not a good argument imo. It eliminates a good class of problems sure. But you can’t eliminate their dependence on shared libraries that many commands also use, and that’s what my point was about.

And I’m sure you can find a whole dictionary’s worth of cases where people shoot themselves in the foot with bash. I don’t deny that’s the case. Bash is not a good language where the programmer is guarded from shooting themselves in the foot as much as possible. The guardrails are loose, and it’s the script writer’s job to guard themselves against it. Is that good for an enterprise scenario, where you may either blow something up, drop a database table, lead to the lost of lives or jobs, etc? Absolutely not. Just want to copy some files around and maybe send it to an internal chat for regular reporting? I don’t see why not.

Bash is not your hammer to hit every possible nail out there. That’s not what I’m proposing at all.

[–] FizzyOrange@programming.dev 0 points 4 hours ago

And I certainly am not proposing that we can abandon robustness.

If you're proposing Bash, then yes you are.

You’ll probably hate this, but you can use set -u to catch unassigned variables.

I actually didn't know that, thanks for the hint! I am forced to use Bash occasionally due to misguided coworkers so this will help at least.

But you can’t eliminate their dependence on shared libraries that many commands also use, and that’s what my point was about.

Not sure what you mean here?

Just want to copy some files around and maybe send it to an internal chat for regular reporting? I don’t see why not.

Well if it's just for a temporary hack and it doesn't matter if it breaks then it's probably fine. Not really what is implied by "production" though.

Also even in that situation I wouldn't use it for two reasons:

  1. "Temporary small script" tends to smoothly morph into "10k line monstrosity that the entire system depends on" with no chance for rewrites. It's best to start in a language that can cope with it.
  2. It isn't really any nicer to use Bash over something like Deno. Like... I don't know why you ever would, given the choice. When you take bug fixing into account Bash is going to be slower and more painful.
[–] JamonBear@sh.itjust.works 4 points 15 hours ago (1 children)

Agreed.

Also gtfobins is a great resource in addition to shellcheck to try to make secure scripts.

For instance I felt upon a script like this recently:

#!/bin/bash
# ... some stuff ...
tar -caf archive.tar.bz2 "$@"

Quotes are OK, shellcheck is happy, but, according to gtfobins, you can abuse tar, so running the script like this: ./test.sh /dev/null --checkpoint=1 --checkpoint-action=exec=/bin/sh ends up spawning an interactive shell...

So you can add up binaries insanity on top of bash's mess.

[–] MonkderVierte@lemmy.ml 1 points 13 hours ago

gtfobins

Meh, most in that list are just "if it has the SUID bit set, it can be used to break out of your security context".

[–] toynbee@lemmy.world 9 points 23 hours ago

Over the last ten - fifteen years, I've written lots of scripts for production in bash. They've all served their purposes (after thorough testing) and not failed. Pretty sure one of my oldest (and biggest) is called temporary_fixes.sh and is still in use today. Another one (admittedly not in production) was partially responsible for getting me my current job, I guess because the interviewers wanted to see what kind of person would solve a coding challenge in bash.

However, I would generally agree that - while bash is good for many things and perhaps even "good enough" - any moderately complex problem is probably better solved using a different language.

"Use the best tool for the job, that the person doing the job is best at." That's my approach.

I will use bash or python dart or whatever the project uses.

[–] Die4Ever@programming.dev 35 points 1 day ago (8 children)

I just don't think bash is good for maintaining the code, debugging, growing the code over time, adding automated tests, or exception handling

load more comments (8 replies)
load more comments
view more: next ›